vmscan.c heuristic adjustment for smaller systems
Subsystems (10)
Marc Singer reported an issue where an embedded ARM system performed poorly due to page replacement potentially prematurely replacing mapped memory where there was very little mapped pagecache in use to begin with.
The following patch attempts to address the issue by using the _maximum_ of vm_swappiness and distress to add to the mapped ratio, so that distress doesn't contribute to swap_tendency until it exceeds vm_swappiness, and afterward the effect is not cumulative.
The intended effect is that swap_tendency should vary in a more jagged way, and not be elevated by distress beyond vm_swappiness until distress exceeds vm_swappiness. For instance, since distress is 100 >> zone->prev_priority, no distinction is made between a vm_swappiness of 50 or a vm_swappiness of 90 given the same mapped_ratio.
Marc Singer has results where this is an improvement, and hopefully can clarify as-needed. Help determining whether this policy change is an improvement for a broader variety of systems would be appreciated.
-- wli
mm/vmscan.c +1 -1
@@ -636,7 +636,7 @@* * A 100% value of vm_swappiness overrides this algorithm altogether. */ - swap_tendency = mapped_ratio /2 + distress + vm_swappiness; + swap_tendency = mapped_ratio /2 +max (distress, vm_swappiness);/* * Now use this metric to decide whether to start moving mapped memory
Quoted 8 lines · William Lee Irwin III
Marc Singer reported an issue where an embedded ARM system performed poorly due to page replacement potentially prematurely replacing mapped memory where there was very little mapped pagecache in use to begin with.
Marc Singer has results where this is an improvement, and hopefully can clarify as-needed. Help determining whether this policy change is an improvement for a broader variety of systems would be appreciated.
I have some numbers to clarify the 'improvement'.
Setup:
ARM922 CPU, 200MHz, 32MiB RAM
NFS mounted rootfs, tcp, hard, v3, 4K blocks
Test application copies 41MiB file and prints the elapsed time
The two scenarios differ only in the setting of /proc/sys/vm/swappiness.
swappiness 60 (default) 0 ------------ -------- elapsed time(s) 52.48 52.9 53.13 52.91 53.13 52.87 52.53 53.03 52.35 53.02
mean 52.72 52.94
I'd say that there is no statistically significant difference between these sets of times. However, after I've run the test program, I run the command "ls -l /proc"
swappiness 60 (default) 0 ------------ -------- elapsed time(s) 18 1 30 1 33 1
This is the problem. Once RAM fills with IO buffers, the kernel's tendency to evict mapped pages ruins interactive performance.
Quoted 7 lines · William Lee Irwin III
Quoted 7 lines
Marc Singer reported an issue where an embedded ARM system performed poorly due to page replacement potentially prematurely replacing mapped memory where there was very little mapped pagecache in use to begin with.
Marc Singer has results where this is an improvement, and hopefully can clarify as-needed. Help determining whether this policy change is an improvement for a broader variety of systems would be appreciated.
Quoted 6 lines · Marc Singer
I have some numbers to clarify the 'improvement'.
Setup:
ARM922 CPU, 200MHz, 32MiB RAM
NFS mounted rootfs, tcp, hard, v3, 4K blocks
Test application copies 41MiB file and prints the elapsed time
The two scenarios differ only in the setting of/proc/sys/vm/swappiness.
This doesn't match your first response. Anyway, this one is gets scrapped. I guess if swappiness solves it, then so much the better.
-- wli
Quoted 17 lines · William Lee Irwin III
Quoted 7 lines · William Lee Irwin III
Quoted 7 lines
Marc Singer reported an issue where an embedded ARM system performed poorly due to page replacement potentially prematurely replacing mapped memory where there was very little mapped pagecache in use to begin with.
Marc Singer has results where this is an improvement, and hopefully can clarify as-needed. Help determining whether this policy change is an improvement for a broader variety of systems would be appreciated.Quoted 6 lines · Marc Singer
I have some numbers to clarify the 'improvement'.
Setup:
ARM922 CPU, 200MHz, 32MiB RAM
NFS mounted rootfs, tcp, hard, v3, 4K blocks
Test application copies 41MiB file and prints the elapsed time
The two scenarios differ only in the setting of/proc/sys/vm/swappiness.
Quoted 2 lines
This doesn't match your first response. Anyway, this one is gets scrapped. I guess if swappiness solves it, then so much the better.
Huh? Where do you see a discrepency? I don't think I claimed that the test program performance changed. The noticeable difference is in interactivity once the page cache fills. IMHO, 30 seconds to do a file listing on /proc is extreme.
Quoted 2 lines · William Lee Irwin III
Quoted 2 lines
This doesn't match your first response. Anyway, this one is gets scrapped. I guess if swappiness solves it, then so much the better.
Quoted 4 lines · Marc Singer
Huh? Where do you see a discrepency? I don't think I claimed that the test program performance changed. The noticeable difference is in interactivity once the page cache fills. IMHO, 30 seconds to do a file listing on /proc is extreme.
Oh, sorry, it was unclear to me that the test changed anything but swappiness (i.e. I couldn't tell they included the patch etc.)
-- wli
Quoted 4 lines · William Lee Irwin III
Quoted 2 lines · William Lee Irwin III
Quoted 2 lines
This doesn't match your first response. Anyway, this one is gets scrapped. I guess if swappiness solves it, then so much the better.
Quoted 8 lines
Quoted 4 lines · Marc Singer
Huh? Where do you see a discrepency? I don't think I claimed that the test program performance changed. The noticeable difference is in interactivity once the page cache fills. IMHO, 30 seconds to do a file listing on /proc is extreme.
Oh, sorry, it was unclear to me that the test changed anything but swappiness (i.e. I couldn't tell they included the patch etc.)
Ah, OK. Now I understand your confusion. Based on the numbers, it is clear that your last patch does exactly the same thing as setting swappiness. It is true that I didn't apply it. Still, I think that your change is worth consideration since setting swappiness to zero is such a blunt solution. I apologize for not making this clear before.
Review
Quoted 11 lines · Marc Singer
I'd say that there is no statistically significant difference between these sets of times. However, after I've run the test program, I run the command "ls -l /proc"
(quoted diff) +0
-0swappiness 60 (default) 0 ------------ -------- elapsed time(s) 18 1 30 1 33 1
How on earth can it take half a minute to list /proc?
Quoted 2 lines
This is the problem. Once RAM fills with IO buffers, the kernel's tendency to evict mapped pages ruins interactive performance.
Is everything here on NFS, or are local filesystemms involved? (What does "mount" say?)
Quoted 6 lines · Andrew Morton
Quoted 5 lines · Marc Singer
I'd say that there is no statistically significant difference between these sets of times. However, after I've run the test program, I run the command "ls -l /proc"
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +0
-0swappiness 60 (default) 0 ------------ -------- elapsed time(s) 18 1 30 1 33 1How on earth can it take half a minute to list /proc?
I've watched the vmscan code at work. The memory pressure is so high that it reclaims mapped pages zealously. The program's code pages are being evicted frequently.
I would like to show a video of the ls -l /proc command. It's remarkable. The program pauses after displaying each line.
Quoted 5 lines
Quoted 2 lines
This is the problem. Once RAM fills with IO buffers, the kernel's tendency to evict mapped pages ruins interactive performance.
Is everything here on NFS, or are local filesystemms involved? (What does "mount" say?)
# mount
rootfs on / type rootfs (rw)
/dev/root on / type nfs (rw,v2,rsize=4096,wsize=4096,hard,udp,nolock,addr=192.168.8.1)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw)I've been wondering if the swappiness isn't a red herring. Is it reasonable that the distress value (in refill_inactive_zones ()) be 50?
Quoted 15 lines · Marc Singer
Quoted 13 lines · Andrew Morton
Quoted 11 lines · Marc Singer
I'd say that there is no statistically significant difference between these sets of times. However, after I've run the test program, I run the command "ls -l /proc"
(quoted diff) +0
-0swappiness 60 (default) 0 ------------ -------- elapsed time(s) 18 1 30 1 33 1
Quoted 5 lines
Quoted 1 line
How on earth can it take half a minute to list /proc?
I've watched the vmscan code at work. The memory pressure is so high that it reclaims mapped pages zealously. The program's code pages are being evicted frequently.
Which tends to imply that the VM is not reclaiming any of that nfs-backed pagecache.
Quoted 3 lines
I've been wondering if the swappiness isn't a red herring. Is it reasonable that the distress value (in
refill_inactive_zones()) be 50?
I'd assume that setting swappiness to zero simply means that you still have all of your libc in pagecache when running ls.
What happens if you do the big file copy, then run `sync', then do the ls?
Have you experimented with the NFS mount options? v2? UDP?
Quoted 1 line · Andrew Morton
What happens if you do the big file copy, then run `sync', then do the ls?
You shouldn't ever need to do "sync" with NFS unless you are using mmap(). close() will suffice to flush out all dirty pages in the case of ordinary file writes.
Cheers,
Trond
Quoted 16 lines · Andrew Morton
Quoted 15 lines · Marc Singer
Quoted 13 lines · Andrew Morton
Quoted 11 lines · Marc Singer
I'd say that there is no statistically significant difference between these sets of times. However, after I've run the test program, I run the command "ls -l /proc"
(quoted diff) +0
-0swappiness 60 (default) 0 ------------ -------- elapsed time(s) 18 1 30 1 33 1
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
How on earth can it take half a minute to list /proc?
I've watched the vmscan code at work. The memory pressure is so high that it reclaims mapped pages zealously. The program's code pages are being evicted frequently.
Which tends to imply that the VM is not reclaiming any of that nfs-backed pagecache.
I don't think that's the whole story. They question is why.
Quoted 6 lines
Quoted 3 lines
I've been wondering if the swappiness isn't a red herring. Is it reasonable that the distress value (in
refill_inactive_zones()) be 50?I'd assume that setting swappiness to zero simply means that you still have all of your libc in pagecache when running ls.
Perhaps. I think it is more important that it is still mapped.
Quoted 2 lines
What happens if you do the big file copy, then run `sync', then do the ls?
It still takes a long time. I'm watching the network load as I perform the ls. There's almost 20 seconds of no screen activity while NFS reloads the code.
Quoted 2 lines
Have you experimented with the NFS mount options? v2? UDP?
Doesn't seem to matter. I've used v2, v3, UDP and TCP.
I have more data.
All of these tests are performed at the console, one command at a time. I have a telnet daemon available, so I open a second connection to the target system. I run a continuous loop of file copies on the console and I execute 'ls -l /proc' in the telnet window. It's a little slow, but it isn't unreasonable. Hmm. I then run the copy command in the telnet window followed by the 'ls -l /proc'. It works fine. I logout of the console session and perform the telnet window test again. The 'ls -l /proc takes 30 seconds.
When there is more than one process running, everything is peachy. When there is only one process (no context switching) I see the slow performance. I had a hypothesis, but my test of that hypothesis failed.
Quoted 13 lines · Marc Singer
Quoted 11 lines · Andrew Morton
Quoted 8 lines · Marc Singer
Quoted 2 lines · Andrew Morton
How on earth can it take half a minute to list /proc?
I've watched the vmscan code at work. The memory pressure is so high that it reclaims mapped pages zealously. The program's code pages are being evicted frequently.
Quoted 6 lines
Quoted 2 lines
Which tends to imply that the VM is not reclaiming any of that nfs-backed pagecache.
I don't think that's the whole story. They question is why.
swappiness is pretty arbitrary and unfortunately it means different things to machines with different sized memory.
Also, once you *have* gone past the reclaim_mapped threshold, mapped pages aren't really given any preference above unmapped pages.
I have a small patchset which splits the active list roughly into mapped and unmapped pages. It might hopefully solve your problem. Would you give it a try? It is pretty stable here.
Nick
Quoted 8 lines · Nick Piggin
swappiness is pretty arbitrary and unfortunately it means different things to machines with different sized memory. Also, once you *have* gone past the
reclaim_mappedthreshold, mapped pages aren't really given any preference above unmapped pages.
I have a small patchset which splits the active list roughly into mapped and unmapped pages. It might hopefully solve your problem. Would you give it a try? It is pretty stable here.
It would be interesting to see the results of this on Marc's system. It's a more comprehensive solution than tweaking numbers.
-- wli
Quoted 12 lines · William Lee Irwin III
Quoted 8 lines · Nick Piggin
swappiness is pretty arbitrary and unfortunately it means different things to machines with different sized memory. Also, once you *have* gone past the
reclaim_mappedthreshold, mapped pages aren't really given any preference above unmapped pages.
I have a small patchset which splits the active list roughly into mapped and unmapped pages. It might hopefully solve your problem. Would you give it a try? It is pretty stable here.
Quoted 3 lines
It would be interesting to see the results of this on Marc's system. It's a more comprehensive solution than tweaking numbers.
Well, here is the current patch against 2.6.5-mm6. -mm is different enough from -linus now that it is not 100% trivial to patch (mainly the rmap and hugepages work).
Marc if you could test this it would be great. I've been doing very swap heavy tests for the last 24 hours on a SMP system here, so it should be fairly stable.
It replaces /proc/sys/vm/swappiness with /proc/sys/vm/mapped_page_cost, which is in units of unmapped pages. I have found 8 to be pretty good, so that is the default. Higher makes it less likely to evict mapped pages.
Nick
linux-2.6-npiggin/include/linux/buffer_head.h | 1 linux-2.6-npiggin/include/linux/mm_inline.h | 48 +++++-- linux-2.6-npiggin/include/linux/mmzone.h | 28 ---- linux-2.6-npiggin/include/linux/page-flags.h | 54 ++++--- linux-2.6-npiggin/include/linux/swap.h | 4 linux-2.6-npiggin/kernel/sysctl.c | 9 - linux-2.6-npiggin/mm/filemap.c | 6 linux-2.6-npiggin/mm/hugetlb.c | 9 - linux-2.6-npiggin/mm/memory.c | 1 linux-2.6-npiggin/mm/page_alloc.c | 26 ++- linux-2.6-npiggin/mm/shmem.c | 1 linux-2.6-npiggin/mm/swap.c | 54 +------ linux-2.6-npiggin/mm/vmscan.c | 176 ++++++++++---------------- 13 files changed, 193 insertions(+), 224 deletions(-)
include/linux/buffer_head.h renamed from include/linux/buffer_head.h~rollup +1 -0
@@ -11,6 +11,7 @@#include <linux/fs.h> #include <linux/linkage.h> #include <linux/wait.h> +#include <linux/mm_inline.h> #include <asm/atomic.h> enum bh_state_bits {
include/linux/mm_inline.h renamed from include/linux/mm_inline.h~rollup +40 -8
69 lines not on this page · raw message
include/linux/mmzone.h renamed from include/linux/mmzone.h~rollup +6 -22
@@ -104,11 +104,14 @@ struct zone {ZONE_PADDING (_pad1_) spinlock_t lru_lock; -struct list_head active_list; +struct list_head active_mapped_list; +struct list_head active_unmapped_list;struct list_head inactive_list; -atomic_t nr_scan_active; +atomic_t nr_scan_active_mapped; +atomic_t nr_scan_active_unmapped;atomic_t nr_scan_inactive; -unsigned long nr_active; +unsigned long nr_active_mapped; +unsigned long nr_active_unmapped;unsigned long nr_inactive;int all_unreclaimable;/* All pages pinned */ unsigned long pages_scanned;/* since last reclaim */
@@ -116,25 +119,6 @@ struct zone {ZONE_PADDING (_pad2_)/* -* prev_priority holds the scanning priority for this zone. It is -* defined as the scanning priority at which we achieved our reclaim -* target at the previous try_to_free_pages() or balance_pgdat() -* invokation. -* -* We use prev_priority as a measure of how much stress page reclaim is -* under - it drives the swappiness decision: whether to unmap mapped -* pages. -* -* temp_priority is used to remember the scanning priority at which -* this zone was successfully refilled to free_pages == pages_high. -* -* Access to both these fields is quite racy even on uniprocessor. But -* it is expected to average out OK. -*/ - int temp_priority; -int prev_priority; - -/* * free areas of different sizes */ struct free_area free_area[MAX_ORDER];
include/linux/page-flags.h renamed from include/linux/page-flags.h~rollup +31 -23
67 lines not on this page · raw message
include/linux/swap.h renamed from include/linux/swap.h~rollup +1 -3
@@ -165,8 +165,6 @@ extern unsigned int nr_free_pagecache_pa/* linux/mm/swap.c */ extern void FASTCALL (lru_cache_add (struct page *));extern void FASTCALL (lru_cache_add_active (struct page *)); -extern void FASTCALL (activate_page (struct page *)); -extern void FASTCALL (mark_page_accessed (struct page *));extern void lru_add_drain (void );extern int rotate_reclaimable_page (struct page *page);extern void swap_setup (void );
@@ -174,7 +172,7 @@ extern void swap_setup(void);/* linux/mm/vmscan.c */ extern int try_to_free_pages (struct zone **,unsigned int ,unsigned int );extern int shrink_all_memory (int ); -extern int vm_swappiness; +extern int vm_mapped_page_cost;#ifdef CONFIG_MMU /* linux/mm/shmem.c */
kernel/sysctl.c renamed from kernel/sysctl.c~rollup +5 -4
@@ -621,6 +621,7 @@ static ctl_table kern_table[] = {/* Constants for minimum and maximum testing in vm_table. We use these as one-element integer vectors. */ static int zero; +static int one =1 ;static int one_hundred =100 ;
@@ -697,13 +698,13 @@ static ctl_table vm_table[] = { }, { .ctl_name = VM_SWAPPINESS, - .procname = "swappiness", - .data = &vm_swappiness, - .maxlen = sizeof(vm_swappiness), + .procname = "mapped_page_cost", + .data = &vm_mapped_page_cost, + .maxlen = sizeof(vm_mapped_page_cost), .mode = 0644, .proc_handler = &proc_dointvec_minmax, .strategy = &sysctl_intvec, - .extra1 = &zero, + .extra1 = &one, .extra2 = &one_hundred, }, #ifdef CONFIG_HUGETLB_PAGE
mm/filemap.c renamed from mm/filemap.c~rollup +1 -5
@@ -663,11 +663,7 @@ page_ok:if (mapping_writably_mapped (mapping))flush_dcache_page (page); -/* -* Mark the page accessed if we read the beginning. -*/ -if (!offset) -mark_page_accessed (page); +mark_page_accessed (page);/* * Ok, we have the page, and it's up-to-date, so
mm/hugetlb.c renamed from mm/hugetlb.c~rollup +6 -3
@@ -127,9 +127,12 @@ static void update_and_free_page(structint i; nr_huge_pages--;for (i =0 ; i < (HPAGE_SIZE /PAGE_SIZE ); i++) { - page[i].flags &= ~(1 << PG_locked |1 << PG_error |1 << PG_referenced | -1 << PG_dirty |1 << PG_active |1 << PG_reserved | -1 << PG_private |1 << PG_writeback); + page[i].flags &= ~( +1 << PG_locked |1 << PG_error | +1 << PG_referenced |1 << PG_dirty | +1 << PG_active_mapped |1 << PG_active_unmapped | +1 << PG_reserved |1 << PG_private | +1 << PG_writeback);set_page_count (&page[i],0 ); }set_page_count (page,1 );
mm/memory.c renamed from mm/memory.c~rollup +1 -0
@@ -38,6 +38,7 @@#include <linux/kernel_stat.h> #include <linux/mm.h> +#include <linux/mm_inline.h> #include <linux/hugetlb.h> #include <linux/mman.h> #include <linux/swap.h>
mm/page_alloc.c renamed from mm/page_alloc.c~rollup +15 -11
@@ -82,7 +82,7 @@ static void bad_page(const char *functio page->flags &= ~(1 << PG_private | 1 << PG_locked | 1 << PG_lru | - 1 << PG_active | + 1 << PG_active_mapped | 1 << PG_dirty | 1 << PG_maplock | 1 << PG_anon |
@@ -224,7 +224,8 @@ static inline void free_pages_check(cons1 << PG_lru |1 << PG_private |1 << PG_locked | -1 << PG_active | +1 << PG_active_mapped | +1 << PG_active_unmapped |1 << PG_reclaim |1 << PG_slab | 1 << PG_maplock |
@@ -334,7 +335,8 @@ static void prep_new_page(struct page *p1 << PG_private |1 << PG_locked |1 << PG_lru | -1 << PG_active | +1 << PG_active_mapped | +1 << PG_active_unmapped |1 << PG_dirty |1 << PG_reclaim | 1 << PG_maplock |
@@ -859,7 +861,8 @@ unsigned int nr_used_zone_pages(void)struct zone *zone;for_each_zone (zone) - pages += zone->nr_active + zone->nr_inactive; + pages += zone->nr_active_mapped + zone->nr_active_unmapped + + zone->nr_inactive;return pages; }
@@ -996,7 +999,7 @@ void get_zone_counts(unsigned long *acti *inactive =0 ; *free =0 ;for_each_zone (zone) { - *active += zone->nr_active; + *active += zone->nr_active_mapped + zone->nr_active_unmapped; *inactive += zone->nr_inactive; *free += zone->free_pages; }
@@ -1114,7 +1117,7 @@ void show_free_areas(void)K (zone->pages_min),K (zone->pages_low),K (zone->pages_high), -K (zone->nr_active), +K (zone->nr_active_mapped + zone->nr_active_unmapped),K (zone->nr_inactive),K (zone->present_pages) );
@@ -1461,8 +1464,6 @@ static void __init free_area_init_core(s zone->zone_pgdat = pgdat; zone->free_pages =0 ; - zone->temp_priority = zone->prev_priority =DEF_PRIORITY ; -/* * The per-cpu-pages pools are set to around 1000th of the * size of the zone. But no more than 1/4 of a meg - there's
@@ -1496,11 +1497,14 @@ static void __init free_area_init_core(s }printk (" %s zone: %lu pages, LIFO batch:%lu\n" , zone_names[j], realsize, batch); -INIT_LIST_HEAD (&zone->active_list); +INIT_LIST_HEAD (&zone->active_mapped_list); +INIT_LIST_HEAD (&zone->active_unmapped_list);INIT_LIST_HEAD (&zone->inactive_list); -atomic_set (&zone->nr_scan_active,0 ); +atomic_set (&zone->nr_scan_active_mapped,0 ); +atomic_set (&zone->nr_scan_active_unmapped,0 );atomic_set (&zone->nr_scan_inactive,0 ); - zone->nr_active =0 ; + zone->nr_active_mapped =0 ; + zone->nr_active_unmapped =0 ; zone->nr_inactive =0 ;if (!size)continue ;
mm/shmem.c renamed from mm/shmem.c~rollup +1 -0
@@ -25,6 +25,7 @@#include <linux/devfs_fs_kernel.h> #include <linux/fs.h> #include <linux/mm.h> +#include <linux/mm_inline.h> #include <linux/mman.h> #include <linux/file.h> #include <linux/swap.h>
mm/swap.c renamed from mm/swap.c~rollup +13 -41
78 lines not on this page · raw message
mm/vmscan.c renamed from mm/vmscan.c~rollup +72 -104
328 lines not on this page · raw message
_
Quoted 13 lines · Nick Piggin
Quoted 12 lines · William Lee Irwin III
Quoted 8 lines · Nick Piggin
swappiness is pretty arbitrary and unfortunately it means different things to machines with different sized memory. Also, once you *have* gone past the
reclaim_mappedthreshold, mapped pages aren't really given any preference above unmapped pages.
I have a small patchset which splits the active list roughly into mapped and unmapped pages. It might hopefully solve your problem. Would you give it a try? It is pretty stable here.
Quoted 7 lines
Quoted 3 lines
It would be interesting to see the results of this on Marc's system. It's a more comprehensive solution than tweaking numbers.
Well, here is the current patch against 2.6.5-mm6. -mm is different enough from -linus now that it is not 100% trivial to patch (mainly the rmap and hugepages work).
Will this work against 2.6.5 without -mm6?
As an aside, I've been using SVN to manage my kernel sources. While I'd be thrilled to make it work, it simply doesn't seem to have the heavy lifting capability to handle the kernel work. I know the rudiments of using BK. What I'd like is some sort of HOWTO with example of common tasks for kernel development. Know of any?
Quoted 3 lines
Marc if you could test this it would be great. I've been doing very swap heavy tests for the last 24 hours on a SMP system here, so it should be fairly stable.
I'm game.
Quoted 4 lines
It replaces
/proc/sys/vm/swappinesswith/proc/sys/vm/mapped_page_cost, which is in units of unmapped pages. I have found 8 to be pretty good, so that is the default. Higher makes it less likely to evict mapped pages.
Sounds good.
Cheers.
Quoted 21 lines · Marc Singer
Quoted 19 lines · Nick Piggin
Quoted 16 lines · William Lee Irwin III
Quoted 8 lines · Nick Piggin
swappiness is pretty arbitrary and unfortunately it means different things to machines with different sized memory. Also, once you *have* gone past the
reclaim_mappedthreshold, mapped pages aren't really given any preference above unmapped pages.
I have a small patchset which splits the active list roughly into mapped and unmapped pages. It might hopefully solve your problem. Would you give it a try? It is pretty stable here.It would be interesting to see the results of this on Marc's system. It's a more comprehensive solution than tweaking numbers.
Quoted 7 lines
Quoted 3 lines
Well, here is the current patch against 2.6.5-mm6. -mm is different enough from -linus now that it is not 100% trivial to patch (mainly the rmap and hugepages work).
Will this work against 2.6.5 without -mm6?
Unfortunately it won't patch easily. If this is a big problem for you I could make you up a 2.6.5 version.
Quoted 6 lines
As an aside, I've been using SVN to manage my kernel sources. While I'd be thrilled to make it work, it simply doesn't seem to have the heavy lifting capability to handle the kernel work. I know the rudiments of using BK. What I'd like is some sort of HOWTO with example of common tasks for kernel development. Know of any?
Well I don't do a great deal of coding or merging, but I use Andrew Morton's patch scripts which make things very easy for me.
Regarding bitkeeper, I have never tried it but there is some help in Documentation/BK-usage/ which might be of use to you.
Quoted 10 lines · Nick Piggin
Quoted 7 lines
Quoted 3 lines
Well, here is the current patch against 2.6.5-mm6. -mm is different enough from -linus now that it is not 100% trivial to patch (mainly the rmap and hugepages work).
Will this work against 2.6.5 without -mm6?
Unfortunately it won't patch easily. If this is a big problem for you I could make you up a 2.6.5 version.
We'll, I'll try applying his patch and then yours. If it doesn't work I'll let you know.
Quoted 11 lines
Quoted 7 lines
As an aside, I've been using SVN to manage my kernel sources. While I'd be thrilled to make it work, it simply doesn't seem to have the heavy lifting capability to handle the kernel work. I know the rudiments of using BK. What I'd like is some sort of HOWTO with example of common tasks for kernel development. Know of any?
Well I don't do a great deal of coding or merging, but I use Andrew Morton's patch scripts which make things very easy for me.
Where does he keep 'em.
Quoted 3 lines
Regarding bitkeeper, I have never tried it but there is some help in
Documentation/BK-usage/which might be of use to you.
I'll read it. Thanks.
Quoted 10 lines · Marc Singer
Quoted 8 lines · Nick Piggin
Quoted 7 lines
Quoted 3 lines
Well, here is the current patch against 2.6.5-mm6. -mm is different enough from -linus now that it is not 100% trivial to patch (mainly the rmap and hugepages work).
Will this work against 2.6.5 without -mm6?
Quoted 7 lines
Quoted 2 lines
Unfortunately it won't patch easily. If this is a big problem for you I could make you up a 2.6.5 version.
We'll, I'll try applying his patch and then yours. If it doesn't work I'll let you know.
OK thanks.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
As an aside, I've been using SVN to manage my kernel sources. While I'd be thrilled to make it work, it simply doesn't seem to have the heavy lifting capability to handle the kernel work. I know the rudiments of using BK. What I'd like is some sort of HOWTO with example of common tasks for kernel development. Know of any?
Quoted 7 lines
Quoted 3 lines
Well I don't do a great deal of coding or merging, but I use Andrew Morton's patch scripts which make things very easy for me.
Where does he keep 'em.
Quoted 5 lines · Nick Piggin
Quoted 3 lines
We'll, I'll try applying his patch and then yours. If it doesn't work I'll let you know.
OK thanks.
There appear to be a lot of conflicts between my development tree and the -mm6 patch. Even your patch doesn't apply cleanly, though I think it is only because a piece has already been applied.
I'm starting with 2.6.5, applying Russell King's 2.6.5 patch from the 8th, applying -mm6 patch, and then yours. It looks like a good bit of Russell's patch has been included in the -mm6. But not enough of mm6 is present in my tree for your patch to work.
I'm working on the scripts and BK docs. At this point, I may have to wait for 2.6.6 before we can make another test.
Quoted 9 lines · Marc Singer
All of these tests are performed at the console, one command at a time. I have a telnet daemon available, so I open a second connection to the target system. I run a continuous loop of file copies on the console and I execute 'ls -l /proc' in the telnet window. It's a little slow, but it isn't unreasonable. Hmm. I then run the copy command in the telnet window followed by the 'ls -l /proc'. It works fine. I logout of the console session and perform the telnet window test again. The 'ls -l /proc takes 30 seconds.
Quoted 4 lines
When there is more than one process running, everything is peachy. When there is only one process (no context switching) I see the slow performance. I had a hypothesis, but my test of that hypothesis failed.
Guys, this tends to indicate that we _must_ have up to date aging information from the PTE - if not, we're liable to miss out on the pressure from user applications. The "lazy" method which 2.4 will allow is not possible with 2.6.
This means we must flush the TLB when we mark the PTE old.
Might be worth reading my thread on linux-mm about this and commenting? (hint hint)
Signature · 4 lines
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 PCMCIA - http://pcmcia.arm.linux.org.uk/
2.6 Serial coreQuoted 3 lines · Marc Singer
Quoted 3 lines
I've watched the vmscan code at work. The memory pressure is so high that it reclaims mapped pages zealously. The program's code pages are being evicted frequently.
Quoted 2 lines · Andrew Morton
Which tends to imply that the VM is not reclaiming any of that nfs-backed pagecache.
The observation that prompted the max() vs. addition was:
On Sat, Apr 17, 2004 at 10:57:24AM -0700, Marc Singer wrote:
> I don't think that's the whole story. I printed distress, > mapped_ratio, and swappiness when vmscan starts trying to reclaim > mapped pages.
> reclaim_mapped: distress 50 mapped_ratio 0 swappiness 60
> 50 + 60 > 100
> So, part of the problem is swappiness. I could set that value to 25, > for example, to stop the machine from swapping.
> I'd be fine stopping here, except for you comment about what > swappiness means. In my case, nearly none of memory is mapped. It is > zone priority which has dropped to 1 that is precipitating the > eviction. Is this what you expect and want?
Quoted 3 lines · Marc Singer
Quoted 3 lines
I've been wondering if the swappiness isn't a red herring. Is it reasonable that the distress value (in
refill_inactive_zones()) be 50?
Quoted 4 lines · Andrew Morton
I'd assume that setting swappiness to zero simply means that you still have all of your libc in pagecache when running ls.
What happens if you do the big file copy, then run `sync', then do the ls? Have you experimented with the NFS mount options? v2? UDP?
I wonder if the ptep_test_and_clear_young() TLB flushing is related.
-- wli
Quoted 8 lines · William Lee Irwin III
Quoted 4 lines · Andrew Morton
I'd assume that setting swappiness to zero simply means that you still have all of your libc in pagecache when running ls.
What happens if you do the big file copy, then run `sync', then do the ls? Have you experimented with the NFS mount options? v2? UDP?I wonder if the
ptep_test_and_clear_young()TLB flushing is related.
That, or page_referenced() always returns true on this ARM implementation or some such silliness. Everything here points at the VM being unable to reclaim that clean pagecache.
Quoted 8 lines · Andrew Morton
Quoted 7 lines · William Lee Irwin III
Quoted 4 lines · Andrew Morton
I'd assume that setting swappiness to zero simply means that you still have all of your libc in pagecache when running ls.
What happens if you do the big file copy, then run `sync', then do the ls? Have you experimented with the NFS mount options? v2? UDP?
Quoted 5 lines
Quoted 1 line
I wonder if the
ptep_test_and_clear_young()TLB flushing is related.That, or
page_referenced()always returns true on this ARM implementation or some such silliness. Everything here points at the VM being unable to reclaim that clean pagecache.
How can I tell? Is it something like this: because page_referenced() always returns true (which I haven't investigated) then the page eviction code cannot distinguish mapped from cache pages and therefore selects valuable, mapped pages.
Quoted 10 lines · Marc Singer
Quoted 8 lines · Andrew Morton
Quoted 7 lines · William Lee Irwin III
Quoted 4 lines · Andrew Morton
I'd assume that setting swappiness to zero simply means that you still have all of your libc in pagecache when running ls.
What happens if you do the big file copy, then run `sync', then do the ls? Have you experimented with the NFS mount options? v2? UDP?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I wonder if the
ptep_test_and_clear_young()TLB flushing is related.That, or
page_referenced()always returns true on this ARM implementation or some such silliness. Everything here points at the VM being unable to reclaim that clean pagecache.How can I tell?
Well some more descriptions of what the system does after that copy-to-nfs would help. Does it _ever_ come good, or is a reboot needed, etc?
What does `vmstat 1' say during the copy, and during the ls?
/proc/vmstats before and after the ls.
Try doing the copy, then when it has finished do the old memset(malloc(24M)) and monitor the `vmstat 1' output while it runs, capture /proc/meminfo before and after.
None of the problems you report are present on x86 as far as I can tell, so...
Quoted 2 lines · Andrew Morton
Well some more descriptions of what the system does after that copy-to-nfs would help. Does it _ever_ come good, or is a reboot needed, etc?
As far as I can tell, it never gets good. I run the same ls command over and over. Ten times. Always the same behavior. Sometimes it gets even slower. I can set swappiness to zero and it responds normally, immediately.
Quoted 1 line
What does `vmstat 1' say during the copy, and during the ls?
I thought I sent a message about this. I've found that the problem *only* occurs when there is exactly one process running. If I open a second console (via telnet) then the slow-down behavior disappears. If I logout of the console session and run the tests from the telnet session then I *do* see the problem again.
Quoted 1 line
/proc/vmstatsbefore and after the ls.
This one I can do.
BEFORE
nr_dirty 0
nr_writeback 0
nr_unstable 0
nr_page_table_pages 49
nr_mapped 163
nr_slab 255
pgpgin 4
pgpgout 0
pswpin 0
pswpout 0
pgalloc_high 0
pgalloc_normal 0
pgalloc_dma 78568
pgfree 79274
pgactivate 8428
pgdeactivate 8324
pgfault 17112
pgmajfault 1348
pgrefill_high 0
pgrefill_normal 0
pgrefill_dma 543296
pgsteal_high 0
pgsteal_normal 0
pgsteal_dma 60834
pgscan_kswapd_high 0
pgscan_kswapd_normal 0
pgscan_kswapd_dma 189700
pgscan_direct_high 0
pgscan_direct_normal 0
pgscan_direct_dma 31746
pginodesteal 0
slabs_scanned 1586
kswapd_steal 30907
kswapd_inodesteal 0
pageoutrun 77994
allocstall 82
pgrotated 0ls -l /proc runs slowly.
AFTER
nr_dirty 0
nr_writeback 0
nr_unstable 0
nr_page_table_pages 49
nr_mapped 164
nr_slab 225
pgpgin 4
pgpgout 0
pswpin 0
pswpout 0
pgalloc_high 0
pgalloc_normal 0
pgalloc_dma 85378
pgfree 86106
pgactivate 11759
pgdeactivate 11650
pgfault 21293
pgmajfault 2316
pgrefill_high 0
pgrefill_normal 0
pgrefill_dma 616785
pgsteal_high 0
pgsteal_normal 0
pgsteal_dma 67511
pgscan_kswapd_high 0
pgscan_kswapd_normal 0
pgscan_kswapd_dma 200241
pgscan_direct_high 0
pgscan_direct_normal 0
pgscan_direct_dma 31746
pginodesteal 0
slabs_scanned 1586
kswapd_steal 37584
kswapd_inodesteal 0
pageoutrun 78405
allocstall 82
pgrotated 0ls -l still slow.
Anything interesting?
Quoted 3 lines
Try doing the copy, then when it has finished do the old memset(malloc(24M)) and monitor the `vmstat 1' output while it runs, capture
/proc/meminfobefore and after.
Here's the meminfo version of the same test above.
BEFORE
MemTotal: 30256 kB MemFree: 3312 kB Buffers: 0 kB Cached: 24512 kB SwapCached: 0 kB Active: 732 kB Inactive: 24084 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 30256 kB LowFree: 3312 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB Mapped: 700 kB Slab: 1024 kB Committed_AS: 476 kB PageTables: 196 kB VmallocTotal: 434176 kB VmallocUsed: 65924 kB VmallocChunk: 368252 kB
ls -l runs slowly
AFTER
MemTotal: 30256 kB MemFree: 3420 kB Buffers: 0 kB Cached: 24448 kB SwapCached: 0 kB Active: 772 kB Inactive: 23984 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 30256 kB LowFree: 3420 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB Mapped: 692 kB Slab: 964 kB Committed_AS: 476 kB PageTables: 196 kB VmallocTotal: 434176 kB VmallocUsed: 65924 kB VmallocChunk: 368252 kB
ls -l still runs slowly.
The copy w/vmstat involves two processes. It doesn't exhibit the problems.
Quoted 2 lines
None of the problems you report are present on x86 as far as I can tell, so...
I don't expect you would. Are you confident that this doesn't happen on a 386 NFS root mounted system in single-user mode? I don't have an IA32 system where I can test this scenario.
Quoted 2 lines · Marc Singer
I thought I sent a message about this. I've found that the problem *only* occurs when there is exactly one process running.
BINGO! ;)
Looks like this could be the referenced bits not being flushed from the MMU and not found by the VM...
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 9 lines · Rik van Riel
Quoted 2 lines · Marc Singer
I thought I sent a message about this. I've found that the problem *only* occurs when there is exactly one process running.
BINGO! ;)
Looks like this could be the referenced bits not being flushed from the MMU and not found by the VM...
Can you be a little more verbose for me? The ARM MMU doesn't keep track of page references, AFAICT. How does a context switch change this?
I have looked into the case where the TLB for an old page isn't being flushed (by design), but I've been unable to fix the problem by forcing a TLB flush whenever a PTE is zeroed.