pgd_free, pmd_free, and pte_free trapping memory.
On a 512 CPU system with 256 numa nodes, we have an application which forks 500 worker threads. During each fork, 32 pages are allocated on the node where the main thread is doing the forks. The child threads then use sched_set_affinity to migrate to a different cpu. After the application exits, we are loosing approx 15,000 pages on main node. If we echo "0 0" >/proc/sys/vm/pagetable_cache, the memory gets returned. This was run on a 2.4 kernel, but the code in question is identical in 2.6.
Note: because of memory size, pagetable_cache sizes are 25 for min and 15559 for max.
Looking through the code, we have identified the source of the problem. The fork is occuring on one cpu where the pgd, pmd, and pte allocations get pages of memory local to that cpu. The worker thread is then migrated to a different cpu where it exits. The pages are then placed on the cpu which is very distant from where the memory is located.
I looked at the i386 code which appears to have been very similar to the ia64 at one point in time, but no longer. They appear to have completely eliminated the quicklists. Is this the right direction for ia64?
Since, when the pgd, pmd, and pte are ready to be freed, they are zeroed out again, I understand the benefit to keeping the entry around to save the time for zeroing out the page again. Why not have a single quicklist where all three are placed. How would node locality best play into placing items on the lists? Should we have one quicklist on each cpu that a cpu returns node local pages and then a node quicklist where we place pages that are not node local using cmpxchg?
One other related but different question. The pagetable_cache size of 15,559 seems a little large. Given that this machine has a large amount of memory, I understand that this doesn't seem too outragously large. What role should node memory play in setting pagetable_cache max size?
As a simple diff to open discussions, I have included the following patch. On our above test, the problem prevents the pages from being trapped. The method is simple, if the page that is being freed is not physically on this node, it is freed, otherwise it is added to the quicklist.
Thanks for your attention,
Robin Holt
include/asm-ia64/pgalloc.h Tue Mar 16 05:12:55 2004 renamed from usr/tmp/TmpDir.9611-0/linux/include/asm-ia64/pgalloc.h_1.15 Tue Mar 16 05:13:05 2004 +19 -0
@@ -18,6 +18,7 @@ #include <linux/compiler.h> #include <linux/mm.h> #include <linux/threads.h> +#include <linux/mmzone.h> #include <asm/mmu_context.h> #include <asm/processor.h>
@@ -65,6 +66,12 @@ static inline void pgd_free (pgd_t *pgd) { +#ifdef CONFIG_NUMA + if(page_zone(virt_to_page(pgd))->zone_pgdat->node_id != numa_node_id()) { + free_page((unsigned long) pgd); + return; + } +#endif *(unsigned long *)pgd = (unsigned long) pgd_quicklist; pgd_quicklist = (unsigned long *) pgd; ++pgtable_cache_size;
@@ -103,6 +110,12 @@ static inline void pmd_free (pmd_t *pmd) { +#ifdef CONFIG_NUMA + if(page_zone(virt_to_page(pmd))->zone_pgdat->node_id != numa_node_id()) { + free_page((unsigned long) pmd); + return; + } +#endif *(unsigned long *)pmd = (unsigned long) pmd_quicklist; pmd_quicklist = (unsigned long *) pmd; ++pgtable_cache_size;
@@ -141,6 +154,12 @@ static inline void pte_free (pte_t *pte) { +#ifdef CONFIG_NUMA + if(page_zone(virt_to_page(pte))->zone_pgdat->node_id != numa_node_id()) { + free_page((unsigned long) pte); + return; + } +#endif *(unsigned long *)pte = (unsigned long) pte_quicklist; pte_quicklist = (unsigned long *) pte; ++pgtable_cache_size;
Quoted 10 lines · Robin Holt
Looking through the code, we have identified the source of the problem. The fork is occuring on one cpu where the pgd, pmd, and pte allocations get pages of memory local to that cpu. The worker thread is then migrated to a different cpu where it exits. The pages are then placed on the cpu which is very distant from where the memory is located.
I looked at the i386 code which appears to have been very similar to the ia64 at one point in time, but no longer. They appear to have completely eliminated the quicklists. Is this the right direction for ia64?
Quoted 7 lines
Since, when the pgd, pmd, and pte are ready to be freed, they are zeroed out again, I understand the benefit to keeping the entry around to save the time for zeroing out the page again. Why not have a single quicklist where all three are placed. How would node locality best play into placing items on the lists? Should we have one quicklist on each cpu that a cpu returns node local pages and then a node quicklist where we place pages that are not node local using cmpxchg?
Tjis quicklist thing is a workaround for not having per-cpu pages in Linux <= 2.4. Your patch is a workaround for a workaround and gets a little ugly. I'd say just rip the quicklists out like x86 and benchmark it.
That's less code and thus less complexity which is always good. Now if the pre-zeroing actually makes a difference we might have to keep small pre-zeroed list around, but I doubt this is really good idea (or even nessecary)
Quoted 19 lines · Christoph Hellwig
Quoted 10 lines · Robin Holt
Looking through the code, we have identified the source of the problem. The fork is occuring on one cpu where the pgd, pmd, and pte allocations get pages of memory local to that cpu. The worker thread is then migrated to a different cpu where it exits. The pages are then placed on the cpu which is very distant from where the memory is located.
I looked at the i386 code which appears to have been very similar to the ia64 at one point in time, but no longer. They appear to have completely eliminated the quicklists. Is this the right direction for ia64?
Quoted 7 lines
Since, when the pgd, pmd, and pte are ready to be freed, they are zeroed out again, I understand the benefit to keeping the entry around to save the time for zeroing out the page again. Why not have a single quicklist where all three are placed. How would node locality best play into placing items on the lists? Should we have one quicklist on each cpu that a cpu returns node local pages and then a node quicklist where we place pages that are not node local using cmpxchg?
Quoted 3 lines
Tjis quicklist thing is a workaround for not having per-cpu pages in Linux <= 2.4. Your patch is a workaround for a workaround and gets a little ugly. I'd say just rip the quicklists out like x86 and benchmark it.
I have a kernel with these ripped out. I have run one simple Aim7 run on a 32P system. The performance fell in the noise range of a normal Aim7 run. Is this a good test to run? Should I focus on any specific benchmark, or run a suite?
Quoted 5 lines
That's less code and thus less complexity which is always good. Now if the pre-zeroing actually makes a difference we might have to keep small pre-zeroed list around, but I doubt this is really good idea (or even nessecary)
The page zeroing costs 4uSec per page (I believe that is the number). With a typical fork taking approx 40 pages, that should be felt during an Aim7 run. It looks like caches are masking some of that out.
Quoted 4 lines · Robin Holt
I have a kernel with these ripped out. I have run one simple Aim7 run on a 32P system. The performance fell in the noise range of a normal Aim7 run. Is this a good test to run? Should I focus on any specific benchmark, or run a suite?
I'm not actually sure. You could ask Ingo Molnar who implemented both the per-cpu pages and ripped out the x86 quicklists if I remember correctly.
Ingo, any idea on how to benchmark that kind of thing best?
Quoted 8 lines
Quoted 4 lines
That's less code and thus less complexity which is always good. Now if the pre-zeroing actually makes a difference we might have to keep small pre-zeroed list around, but I doubt this is really good idea (or even nessecary)
The page zeroing costs 4uSec per page (I believe that is the number). With a typical fork taking approx 40 pages, that should be felt during an Aim7 run. It looks like caches are masking some of that out.
OTOH you have more pages avalilable, the real per-cpu pages have a better cache locality than the quicklists, the kernel has a smaller icache footprint, etc..
Quoted 7 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Robin Holt
Robin> I have a kernel with these ripped out. I have run one simple Robin> Aim7 run on a 32P system. The performance fell in the noise Robin> range of a normal Aim7 run. Is this a good test to run? Robin> Should I focus on any specific benchmark, or run a suite?
Quoted 5 lines
For this kind of test, I'd rather see a micro-benchmark to establish the worst-case effect. Then we have an upper bound and can make a judgment as to whether or not that's a likely scenario. And yes, this would mean that you'd probably want to measure it on a relative small system (certainly not NUMA, perhaps just UP).
I am currently looking into Christoph's suggestion of using a per cpu page allocator. Unfortunately, yesterday was a little busy and I didn't get much of a chance to look into it.
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
That's less code and thus less complexity which is always good.
Now if the pre-zeroing actually makes a difference we might have to keep small pre-zeroed list around, but I doubt this is really good idea (or even nessecary)
Quoted 7 lines
Robin> The page zeroing costs 4uSec per page (I believe that is the Robin> number). With a typical fork taking approx 40 pages, that Robin> should be felt during an Aim7 run. It looks like caches are Robin> masking some of that out.
Try UP. Also, what if the page-size is 64KB? In any case, 4usec is a lot.
I got the 4uSec from Jack Steiner. I don't know if he tested it with 64KB pages. I will check.
Quoted 11 lines
Quoted 8 lines
Robin> The page zeroing costs 4uSec per page (I believe that is the Robin> number). With a typical fork taking approx 40 pages, that Robin> should be felt during an Aim7 run. It looks like caches are Robin> masking some of that out.
Try UP. Also, what if the page-size is 64KB? In any case, 4usec is a lot.
I got the 4uSec from Jack Steiner. I don't know if he tested it with 64KB pages. I will check.
I see the following times for the kernel "clearpage" routine. I run this code in user space but I use the kernel assembly code for clearing pages. AFAICT, the timing should be identical to running it in the kernel.
The test harness ensures that the timing is done with warm TLBs & cold cache (data not in cpu caches):
3.1 usec 16K node local memory
6.1 usec 16K remote memory
12.5 usec 64K node local memory
24.5 usec 64K remote memory
I ran this on Itanium 2 1300MHz cpus. However, processor core speed does not significantly affect timings since most time is spent waiting for off chip memory access.
Also, timing will obviously vary across different platforms.
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.
Quoted 6 lines · Jack Steiner
The test harness ensures that the timing is done with warm TLBs & cold cache (data not in cpu caches):
3.1 usec 16K node local memory
6.1 usec 16K remote memory
Quoted 6 lines
12.5 usec 64K node local memory
24.5 usec 64K remote memoryI ran this on Itanium 2 1300MHz cpus. However, processor core speed does not significantly affect timings since most time is spent waiting for off chip memory access.
Well, pages on the per-cpu list are supposed to be still cache hot..
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Jack Steiner
Jack> 3.1 usec 16K node local memory
Jack> 6.1 usec 16K remote memory
Jack> 12.5 usec 64K node local memory
Jack> 24.5 usec 64K remote memory
Roughly 4.9GByte/sec for local memory. That sounds about right.
Robin: I certainly agree that per-CPU allocation is the Right Thing for SMP. My point is that the quick-list optimization is orthogonal and my gut feeling is that it's worth doing it. LMbench on a UP with and without quicklist should be able to confirm that (and no, I don't care at all how or where the quicklist optimization is implemented---just as long as you can take advantage of the fact that freed directories are already zeroed).
--david
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Jack Steiner
The test harness ensures that the timing is done with warm TLBs & cold cache (data not in cpu caches):
3.1 usec 16K node local memory
6.1 usec 16K remote memory
Quoted 8 lines
Quoted 6 lines
12.5 usec 64K node local memory
24.5 usec 64K remote memoryI ran this on Itanium 2 1300MHz cpus. However, processor core speed does not significantly affect timings since most time is spent waiting for off chip memory access.
Well, pages on the per-cpu list are supposed to be still cache hot..
For some workloads that is true.
The workload that Robin mentioned in the original mail allocated 32 PT pages on each fork. Even if the pages were recently used, in many cases a large portion of those pages are probably not cache-resident.
Signature · 4 lines
Thanks Jack Steiner ([email protected]) 651-683-5302 Principal Engineer SGI - Silicon Graphics, Inc.