Fine granularity page reclaim

DISCUSSION Zheng Liu 3 participants 9 messages Feb 17, 2012 – Apr 7, 2012

Subsystems (3)

Memory management Scheduler MM rmap

lore

Share & tools

lore

Zheng Liu

author

2012-02-17 09:17 UTC #0 lore raw

Hi all,

Currently, we encounter a problem about page reclaim. In our product system, there is a lot of applictions that manipulate a number of files. In these files, they can be divided into two categories. One is index file, another is block file. The number of index files is about 15,000, and the number of block files is about 23,000 in a 2TB disk. The application accesses index file using mmap(2), and read/write block file using pread(2)/pwrite(2). We hope to hold index file in memory as much as possible, and it works well in Redhat 2.6.18-164. It is about 60-70% of index files that can be hold in memory. However, it doesn't work well in Redhat 2.6.32-133. I know in 2.6.18 that the linux uses an active list and an inactive list to handle page reclaim, and in 2.6.32 that they are divided into anonymous list and file list. So I am curious about why most of index files can be hold in 2.6.18? The index file should be replaced because mmap doesn't impact the lru list.

BTW, I have some problems that need to be discussed.

1. I want to let index and block files are separately reclaimed. Is there any ways to satisify me in current upstream?

2. Maybe we can provide a mechansim to let different files to be mapped into differnet nodes. we can provide a ioctl(2) to tell kernel that this file should be mapped into a specific node id. A nid member is added into addpress_space struct. When alloc_page is called, the page can be allocated from that specific node id.

3. Currently the page can be reclaimed according to pid in memcg. But it is too coarse. I don't know whether memcg could provide a fine granularity page reclaim mechansim. For example, the page is reclaimed according to inode number.

I don't subscribe this mailing list, So please Cc me. Thank you.

Regards,
Zheng

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Konstantin Khlebnikov

2012-02-17 20:20 UTC #1 lore raw

Quoted 15 lines · Zheng Liu

Hi all,
Currently, we encounter a problem about page reclaim. In our product system, there is a lot of applictions that manipulate a number of files. In these files, they can be divided into two categories. One is index file, another is block file. The number of index files is about 15,000, and the number of block files is about 23,000 in a 2TB disk. The application accesses index file using mmap(2), and read/write block file using pread(2)/pwrite(2). We hope to hold index file in memory as much as possible, and it works well in Redhat 2.6.18-164. It is about 60-70% of index files that can be hold in memory. However, it doesn't work well in Redhat 2.6.32-133. I know in 2.6.18 that the linux uses an active list and an inactive list to handle page reclaim, and in 2.6.32 that they are divided into anonymous list and file list. So I am curious about why most of index files can be hold in 2.6.18? The index file should be replaced because mmap doesn't impact the lru list.

There was my patch for fixing similar problem with shared/executable mapped pages "vmscan: promote shared file mapped pages" commit 34dbc67a644f and commit c909e99364c maybe it will help in your case.

Quoted 21 lines

BTW, I have some problems that need to be discussed.
1. I want to let index and block files are separately reclaimed. Is there any ways to satisify me in current upstream?
2. Maybe we can provide a mechansim to let different files to be mapped into differnet nodes. we can provide a ioctl(2) to tell kernel that this file should be mapped into a specific node id. A nid member is added into addpress_space struct. When alloc_page is called, the page can be allocated from that specific node id.
3. Currently the page can be reclaimed according to pid in memcg. But it is too coarse. I don't know whether memcg could provide a fine granularity page reclaim mechansim. For example, the page is reclaimed according to inode number.
I don't subscribe this mailing list, So please Cc me. Thank you.
Regards,
Zheng

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email:<a href=mailto:"[email protected]"> [email protected]</a>

Zheng Liu

author

2012-02-20 06:15 UTC #2 lore raw

Cc linux-kernel mailing list.

Quoted 17 lines · Konstantin Khlebnikov

Quoted 15 lines · Zheng Liu
Hi all,
Currently, we encounter a problem about page reclaim. In our product system, there is a lot of applictions that manipulate a number of files. In these files, they can be divided into two categories. One is index file, another is block file. The number of index files is about 15,000, and the number of block files is about 23,000 in a 2TB disk. The application accesses index file using mmap(2), and read/write block file using pread(2)/pwrite(2). We hope to hold index file in memory as much as possible, and it works well in Redhat 2.6.18-164. It is about 60-70% of index files that can be hold in memory. However, it doesn't work well in Redhat 2.6.32-133. I know in 2.6.18 that the linux uses an active list and an inactive list to handle page reclaim, and in 2.6.32 that they are divided into anonymous list and file list. So I am curious about why most of index files can be hold in 2.6.18? The index file should be replaced because mmap doesn't impact the lru list.

Quoted 3 lines

There was my patch for fixing similar problem with shared/executable mapped pages "vmscan: promote shared file mapped pages" commit 34dbc67a644f and commit c909e99364c maybe it will help in your case.

Hi Konstantin,

Thank you for your reply. I have tested it in upstream kernel. These patches are useful for multi-processes applications. But, in our product system, there are some applications that are multi-thread. So 'references_ptes > 1' cannot help these applications to hold the data in memory.

Regards,
Zheng

Quoted 22 lines

Quoted 22 lines
BTW, I have some problems that need to be discussed.
1. I want to let index and block files are separately reclaimed. Is there any ways to satisify me in current upstream?
2. Maybe we can provide a mechansim to let different files to be mapped into differnet nodes. we can provide a ioctl(2) to tell kernel that this file should be mapped into a specific node id. A nid member is added into addpress_space struct. When alloc_page is called, the page can be allocated from that specific node id.
3. Currently the page can be reclaimed according to pid in memcg. But it is too coarse. I don't know whether memcg could provide a fine granularity page reclaim mechansim. For example, the page is reclaimed according to inode number.
I don't subscribe this mailing list, So please Cc me. Thank you.
Regards,
Zheng

Quoted 7 lines

Quoted 6 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email:<a href=mailto:"[email protected]"> [email protected]</a>

Zheng Liu

author

2012-02-20 06:19 UTC #3 lore raw

---------- Forwarded message ----------
From: Zheng Liu <[email protected]>
Date: Mon, Feb 20, 2012 at 2:20 PM
Subject: Re: Fine granularity page reclaim
To: Konstantin Khlebnikov <[email protected]>
Cc: "[email protected]" <[email protected]>, [email protected]

Cc linux-kernel mailing list.

Quoted 24 lines · Konstantin Khlebnikov

Quoted 14 lines · Zheng Liu
Hi all,
Currently, we encounter a problem about page reclaim. In our product system,
there is a lot of applictions that manipulate a number of files. In these files, they can be divided into two categories. One is index file, another is
block file. The number of index files is about 15,000, and the number of block files is about 23,000 in a 2TB disk. The application accesses index file using mmap(2), and read/write block file using pread(2)/pwrite(2). We hope
to hold index file in memory as much as possible, and it works well in Redhat
2.6.18-164. It is about 60-70% of index files that can be hold in memory.
Quoted 8 lines
However, it doesn't work well in Redhat 2.6.32-133. I know in 2.6.18 that the
linux uses an active list and an inactive list to handle page reclaim, and in
2.6.32 that they are divided into anonymous list and file list. So I am curious about why most of index files can be hold in 2.6.18? The index file
should be replaced because mmap doesn't impact the lru list.

Quoted 5 lines

There was my patch for fixing similar problem with shared/executable mapped pages
"vmscan: promote shared file mapped pages" commit 34dbc67a644f and commit c909e99364c
maybe it will help in your case.

Hi Konstantin,

Regards,
Zheng

Quoted 37 lines

Quoted 36 lines
BTW, I have some problems that need to be discussed.
1. I want to let index and block files are separately reclaimed. Is there any
ways to satisify me in current upstream?
2. Maybe we can provide a mechansim to let different files to be mapped into
differnet nodes. we can provide a ioctl(2) to tell kernel that this file should
be mapped into a specific node id. A nid member is added into addpress_space
struct. When alloc_page is called, the page can be allocated from that specific
node id.
3. Currently the page can be reclaimed according to pid in memcg. But it is too
coarse. I don't know whether memcg could provide a fine granularity page reclaim mechansim. For example, the page is reclaimed according to inode number.
I don't subscribe this mailing list, So please Cc me. Thank you.
Regards,
Zheng
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign
http://stopthemeter.ca/
Don't email:<a href=mailto:"[email protected]"> [email protected]</a>

Konstantin Khlebnikov

2012-02-20 07:09 UTC #4 lore raw

In reply to Zheng Liu

Quoted 24 lines · Zheng Liu

Cc linux-kernel mailing list.
Quoted 17 lines · Konstantin Khlebnikov
Quoted 15 lines · Zheng Liu
Hi all,
Currently, we encounter a problem about page reclaim. In our product system, there is a lot of applictions that manipulate a number of files. In these files, they can be divided into two categories. One is index file, another is block file. The number of index files is about 15,000, and the number of block files is about 23,000 in a 2TB disk. The application accesses index file using mmap(2), and read/write block file using pread(2)/pwrite(2). We hope to hold index file in memory as much as possible, and it works well in Redhat 2.6.18-164. It is about 60-70% of index files that can be hold in memory. However, it doesn't work well in Redhat 2.6.32-133. I know in 2.6.18 that the linux uses an active list and an inactive list to handle page reclaim, and in 2.6.32 that they are divided into anonymous list and file list. So I am curious about why most of index files can be hold in 2.6.18? The index file should be replaced because mmap doesn't impact the lru list.
Quoted 3 lines
There was my patch for fixing similar problem with shared/executable mapped pages "vmscan: promote shared file mapped pages" commit 34dbc67a644f and commit c909e99364c maybe it will help in your case.

Quoted 7 lines

Hi Konstantin,
Thank you for your reply. I have tested it in upstream kernel. These patches are useful for multi-processes applications. But, in our product system, there are some applications that are multi-thread. So 'references_ptes> 1' cannot help these applications to hold the data in memory.

Ok, what if you mmap you data as executable, just to test. Then these pages will be activated after first touch. In attachment patch with per-mm flag with the same effect.

Quoted 33 lines

Regards,
Zheng
Quoted 29 lines
Quoted 28 lines
BTW, I have some problems that need to be discussed.
1. I want to let index and block files are separately reclaimed. Is there any ways to satisify me in current upstream?
2. Maybe we can provide a mechansim to let different files to be mapped into differnet nodes. we can provide a ioctl(2) to tell kernel that this file should be mapped into a specific node id. A nid member is added into addpress_space struct. When alloc_page is called, the page can be allocated from that specific node id.
3. Currently the page can be reclaimed according to pid in memcg. But it is too coarse. I don't know whether memcg could provide a fine granularity page reclaim mechansim. For example, the page is reclaimed according to inode number.
I don't subscribe this mailing list, So please Cc me. Thank you.
Regards,
Zheng
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email:<a href=mailto:"[email protected]"> [email protected]</a>

mm: introduce MMF_VM_PREFERRDED flag

From: Konstantin Khlebnikov <[email protected]>

This patch introduce mm->flags bit: MMF_VM_PREFERRED, which doubles access bit weight for this mm.

Currently the only one effect:
mm with this bit activates mapped file pages after first touch, if vma does not marked as sequentially accessed.

This should be per-vma sign, but there no free bits in vma->vm_flags, maybe we can make this stuff 64-only.

interface:
prctl(PR_SET_MM_PREFERRED, 1) to set and
prctl(PR_SET_MM_PREFERRED, 0) to clear.

Signed-off-by Konstantin Khlebnikov <[email protected]>

---
 include/linux/prctl.h |    2 ++
 include/linux/sched.h |    1 +
 kernel/sys.c          |   17 +++++++++++++++++
 mm/rmap.c             |    5 ++++-
 4 files changed, 24 insertions(+), 1 deletions(-)

include/linux/prctl.h +2 -0

index 7ddc7f1..d0f9ceb 100644
@@ -114,4 +114,6 @@
 # define PR_SET_MM_START_BRK		6
 # define PR_SET_MM_BRK			7
 
+#define PR_SET_MM_PREFERRED	36
+
 #endif /* _LINUX_PRCTL_H */

include/linux/sched.h +1 -0

index 75c15c5..b60883a 100644
@@ -437,6 +437,7 @@ extern int get_dumpable(struct mm_struct *mm);
 					/* leave room for more dump flags */
 #define MMF_VM_MERGEABLE	16	/* KSM may merge identical pages */
 #define MMF_VM_HUGEPAGE		17	/* set when VM_HUGEPAGE is set on vma */
+#define MMF_VM_PREFERRED	18	/* Double pte access bits weight */
 
 #define MMF_INIT_MASK		(MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK)

kernel/sys.c +17 -0

index 4070153..bacf8d5 100644
@@ -1810,6 +1810,20 @@ static int prctl_set_mm(int opt, unsigned long addr,
 }
 #endif
 
+static int set_mm_preferred(struct mm_struct *mm, int state)
+{
+	switch (state) {
+		case 0:
+			clear_bit(MMF_VM_PREFERRED, &mm->flags);
+			return 0;
+		case 1:
+			set_bit(MMF_VM_PREFERRED, &mm->flags);
+			return 0;
+		default:
+			return -EINVAL;
+	}
+}
+
 SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 		unsigned long, arg4, unsigned long, arg5)
 {

@@ -1962,6 +1976,9 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 		case PR_SET_MM:
 			error = prctl_set_mm(arg2, arg3, arg4, arg5);
 			break;
+		case PR_SET_MM_PREFERRED:
+			error = set_mm_preferred(me->mm, arg2);
+			break;
 		default:
 			error = -EINVAL;
 			break;

mm/rmap.c +4 -1

index 78cc46b..b0fd1d1 100644
@@ -766,8 +766,11 @@ int page_referenced_one(struct page *page, struct vm_area_struct *vma,
 
 	(*mapcount)--;
 
-	if (referenced)
+	if (referenced) {
+		if (test_bit(MMF_VM_PREFERRED, &mm->flags))
+			referenced <<= 1;
 		*vm_flags |= vma->vm_flags;
+	}
 out:
 	return referenced;
 }

Zheng Liu

author

2012-03-07 17:45 UTC #5 lore raw

Quoted 11 lines · Konstantin Khlebnikov

Quoted 10 lines · Zheng Liu
Cc linux-kernel mailing list.
Quoted 6 lines · Konstantin Khlebnikov
Quoted 4 lines · Zheng Liu
Hi all,
Currently, we encounter a problem about page reclaim. In our product

system,

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
there is a lot of applictions that manipulate a number of files. In

these

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
files, they can be divided into two categories. One is index file,

another is

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
block file. The number of index files is about 15,000, and the number

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
block files is about 23,000 in a 2TB disk. The application accesses

index

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
file using mmap(2), and read/write block file using

pread(2)/pwrite(2). We hope

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
to hold index file in memory as much as possible, and it works well in

Redhat

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
2.6.18-164. It is about 60-70% of index files that can be hold in

memory.

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
However, it doesn't work well in Redhat 2.6.32-133. I know in 2.6.18

that the

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
linux uses an active list and an inactive list to handle page reclaim,

and in

Quoted 2 lines

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2.6.32 that they are divided into anonymous list and file list. So I am curious about why most of index files can be hold in 2.6.18? The index

file

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 1 line
should be replaced because mmap doesn't impact the lru list.
There was my patch for fixing similar problem with shared/executable

mapped pages

Quoted 1 line

Quoted 1 line
Quoted 1 line
"vmscan: promote shared file mapped pages" commit 34dbc67a644f and

commit c909e99364c

Quoted 10 lines

Quoted 9 lines
Quoted 1 line
maybe it will help in your case.
Hi Konstantin,
Thank you for your reply. I have tested it in upstream kernel. These patches are useful for multi-processes applications. But, in our product system, there are some applications that are multi-thread. So 'references_ptes> 1' cannot help these applications to hold the data in memory.

Quoted 4 lines

Ok, what if you mmap you data as executable, just to test. Then these pages will be activated after first touch. In attachment patch with per-mm flag with the same effect.

Hi Konstantin,

Sorry for the delay reply. Last two weeks I was trying these two solutions and evaluating the impacts for the performance in our product system. Good news is that these two solutions both work well. They can keep mapped files in memory under mult-thread. But I have a question for the first solution (map the file with PROT_EXEC flag). I think this way is too tricky. As I said previously, these files that needs to be mapped only are normal index file, and they shouldn't be mapped with PROT_EXEC flag from the view of an application programmer. So actually the key issue is that we should provide a mechanism, which lets different file sets can be reclaimed separately. I am not sure whether this idea is useful or not. So any feedbacks are welcomed.:-). Thank you.

Regards,
Zheng

Quoted 9 lines

Quoted 9 lines
Regards,
Zheng
Quoted 5 lines
Quoted 5 lines
BTW, I have some problems that need to be discussed.
1. I want to let index and block files are separately reclaimed. Is

there any

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
ways to satisify me in current upstream?
2. Maybe we can provide a mechansim to let different files to be

mapped into

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
differnet nodes. we can provide a ioctl(2) to tell kernel that this

file should

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
be mapped into a specific node id. A nid member is added into

addpress_space

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
struct. When alloc_page is called, the page can be allocated from that

specific

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
node id.
3. Currently the page can be reclaimed according to pid in memcg. But

it is too

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
coarse. I don't know whether memcg could provide a fine granularity

page

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
reclaim mechansim. For example, the page is reclaimed according to

inode number.

Quoted 11 lines

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I don't subscribe this mailing list, So please Cc me. Thank you.
Regards,
Zheng
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign

http://stopthemeter.ca/

Quoted 4 lines

Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Don't email:<a href=mailto:"[email protected]"> [email protected]</a>

Konstantin Khlebnikov

2012-03-07 20:33 UTC #6 lore raw

Quoted 46 lines · Zheng Liu

Quoted 36 lines · On Monday, February 20, 2012, Konstantin Khlebnikov <[email protected]…
Quoted 27 lines · Zheng Liu
Cc linux-kernel mailing list.
Quoted 19 lines · Konstantin Khlebnikov
Quoted 16 lines · Zheng Liu
Hi all,
Currently, we encounter a problem about page reclaim. In our product system, there is a lot of applictions that manipulate a number of files. In these files, they can be divided into two categories. One is index file, another is block file. The number of index files is about 15,000, and the number of block files is about 23,000 in a 2TB disk. The application accesses index file using mmap(2), and read/write block file using pread(2)/pwrite(2). We hope to hold index file in memory as much as possible, and it works well in Redhat 2.6.18-164. It is about 60-70% of index files that can be hold in memory. However, it doesn't work well in Redhat 2.6.32-133. I know in 2.6.18 that the linux uses an active list and an inactive list to handle page reclaim, and in 2.6.32 that they are divided into anonymous list and file list. So I am curious about why most of index files can be hold in 2.6.18? The index file should be replaced because mmap doesn't impact the lru list.
Quoted 3 lines
There was my patch for fixing similar problem with shared/executable mapped pages "vmscan: promote shared file mapped pages" commit 34dbc67a644f and commit c909e99364c maybe it will help in your case.
Quoted 7 lines
Hi Konstantin,
Thank you for your reply. I have tested it in upstream kernel. These patches are useful for multi-processes applications. But, in our product system, there are some applications that are multi-thread. So 'references_ptes> 1' cannot help these applications to hold the data in memory.
Quoted 4 lines
Ok, what if you mmap you data as executable, just to test. Then these pages will be activated after first touch. In attachment patch with per-mm flag with the same effect.
Hi Konstantin,

Quoted 12 lines

Sorry for the delay reply. Last two weeks I was trying these two solutions and evaluating the impacts for the performance in our product system. Good news is that these two solutions both work well. They can keep mapped files in memory under mult-thread. But I have a question for the first solution (map the file with PROT_EXEC flag). I think this way is too tricky. As I said previously, these files that needs to be mapped only are normal index file, and they shouldn't be mapped with PROT_EXEC flag from the view of an application programmer. So actually the key issue is that we should provide a mechanism, which lets different file sets can be reclaimed separately. I am not sure whether this idea is useful or not. So any feedbacks are welcomed.:-). Thank you.

Sounds good. Yes, PROT_EXEC isn't very usable and secure, per-mm flag not very flexible too. I prefer setting some kind of memory pressure priorities for each vma and inode. Probably we can sort vma and inodes into different cgroup-like sets and balance memory pressure between them. Maybe someone was thought about it...

Zheng Liu

author

2012-03-08 02:49 UTC #7 lore raw

Quoted 60 lines · Konstantin Khlebnikov

Quoted 46 lines · Zheng Liu
Quoted 36 lines · On Monday, February 20, 2012, Konstantin Khlebnikov <[email protected]…
Quoted 27 lines · Zheng Liu
Cc linux-kernel mailing list.
Quoted 19 lines · Konstantin Khlebnikov
Quoted 16 lines · Zheng Liu
Hi all,
Currently, we encounter a problem about page reclaim. In our product system, there is a lot of applictions that manipulate a number of files. In these files, they can be divided into two categories. One is index file, another is block file. The number of index files is about 15,000, and the number of block files is about 23,000 in a 2TB disk. The application accesses index file using mmap(2), and read/write block file using pread(2)/pwrite(2). We hope to hold index file in memory as much as possible, and it works well in Redhat 2.6.18-164. It is about 60-70% of index files that can be hold in memory. However, it doesn't work well in Redhat 2.6.32-133. I know in 2.6.18 that the linux uses an active list and an inactive list to handle page reclaim, and in 2.6.32 that they are divided into anonymous list and file list. So I am curious about why most of index files can be hold in 2.6.18? The index file should be replaced because mmap doesn't impact the lru list.
Quoted 3 lines
There was my patch for fixing similar problem with shared/executable mapped pages "vmscan: promote shared file mapped pages" commit 34dbc67a644f and commit c909e99364c maybe it will help in your case.
Quoted 7 lines
Hi Konstantin,
Thank you for your reply. I have tested it in upstream kernel. These patches are useful for multi-processes applications. But, in our product system, there are some applications that are multi-thread. So 'references_ptes> 1' cannot help these applications to hold the data in memory.
Quoted 4 lines
Ok, what if you mmap you data as executable, just to test. Then these pages will be activated after first touch. In attachment patch with per-mm flag with the same effect.
Hi Konstantin,
Quoted 12 lines
Sorry for the delay reply. Last two weeks I was trying these two solutions and evaluating the impacts for the performance in our product system. Good news is that these two solutions both work well. They can keep mapped files in memory under mult-thread. But I have a question for the first solution (map the file with PROT_EXEC flag). I think this way is too tricky. As I said previously, these files that needs to be mapped only are normal index file, and they shouldn't be mapped with PROT_EXEC flag from the view of an application programmer. So actually the key issue is that we should provide a mechanism, which lets different file sets can be reclaimed separately. I am not sure whether this idea is useful or not. So any feedbacks are welcomed.:-). Thank you.

Quoted 5 lines

Sounds good. Yes, PROT_EXEC isn't very usable and secure, per-mm flag not very flexible too. I prefer setting some kind of memory pressure priorities for each vma and inode. Probably we can sort vma and inodes into different cgroup-like sets and balance memory pressure between them. Maybe someone was thought about it...

Thanks for your advices. About setting pressure priorities for each vma and inode, I will send a new mail to mailing list to discuss this problem. Maybe someone has some good ideas for it. ;-)

Regards,
Zheng

Ying Han

2012-04-07 00:18 UTC #8 lore raw

In reply to Zheng Liu

Quoted 14 lines · Zheng Liu

Hi all,
Currently, we encounter a problem about page reclaim. In our product system, there is a lot of applictions that manipulate a number of files. In these files, they can be divided into two categories. One is index file, another is block file. The number of index files is about 15,000, and the number of block files is about 23,000 in a 2TB disk. The application accesses index file using mmap(2), and read/write block file using pread(2)/pwrite(2). We hope to hold index file in memory as much as possible, and it works well in Redhat 2.6.18-164. It is about 60-70% of index files that can be hold in memory. However, it doesn't work well in Redhat 2.6.32-133. I know in 2.6.18 that the linux uses an active list and an inactive list to handle page reclaim, and in 2.6.32 that they are divided into anonymous list and file list. So I am curious about why most of index files can be hold in 2.6.18?

One of changes after the split-lru is different scan ratio (active vs inactive) for file-lru and anon-lru. You can check the following two functions:

inactive_anon_is_low_global()
inactive_file_is_low_global()

Depends on your machine size, we might end of scanning more pages on file lru.

--Ying

The index file

Quoted 22 lines

should be replaced because mmap doesn't impact the lru list.
BTW, I have some problems that need to be discussed.
1. I want to let index and block files are separately reclaimed. Is there any ways to satisify me in current upstream?
2. Maybe we can provide a mechansim to let different files to be mapped into differnet nodes. we can provide a ioctl(2) to tell kernel that this file should be mapped into a specific node id. A nid member is added into addpress_space struct. When alloc_page is called, the page can be allocated from that specific node id.
3. Currently the page can be reclaimed according to pid in memcg. But it is too coarse. I don't know whether memcg could provide a fine granularity page reclaim mechansim. For example, the page is reclaimed according to inode number.
I don't subscribe this mailing list, So please Cc me. Thank you.
Regards,
Zheng

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>