ia64 get_mmu_context patch
Gentleman,
Please find attached IA64 context_id patch and supporting data for your Review and consideration.
Regards,
pete
Signed-off-by:
include/asm-ia64/mmu_context.h +3 -0
@@ -34,11 +34,13 @@unsigned int next;/* next context number to use */ unsigned int limit;/* next >= limit => must call wrap_mmu_context() */ unsigned int max_ctx;/* max. context value supported by all CPUs */ +unsigned long *bitmap;/* bitmap size is max_ctx+1 */ };extern struct ia64_ctx ia64_ctx;DECLARE_PER_CPU (u8, ia64_need_tlb_flush); +extern void mmu_context_init (void );extern void wrap_mmu_context (struct mm_struct *mm); static inline void
@@ -86,6 +88,7 @@if (ia64_ctx.next >= ia64_ctx.limit)wrap_mmu_context (mm); mm->context = context = ia64_ctx.next++; +set_bit (context, ia64_ctx.bitmap); } }spin_unlock_irqrestore (&ia64_ctx.lock, flags);
include/asm-ia64/tlbflush.h +1 -0
@@ -51,6 +51,7 @@if (!mm)return ; +clear_bit (mm->context, ia64_ctx.bitmap); mm->context =0 ;if (atomic_read (&mm->mm_users) ==0 )
arch/ia64/mm/tlb.c +37 -21
90 lines not on this page · raw message
arch/ia64/kernel/setup.c +7 -2
@@ -419,6 +419,7 @@#endif cpu_init ();/* initialize the bootstrap CPU */ +mmu_context_init ();/* initialize context_id bitmap */ #ifdef CONFIG_ACPI acpi_boot_init ();
@@ -798,9 +799,13 @@#endif /* set ia64_ctx.max_rid to the maximum RID that is supported by all CPUs: */ -if (ia64_pal_vm_summary (NULL , &vmi) ==0 ) +if (ia64_pal_vm_summary (NULL , &vmi) ==0 ) { max_ctx = (1U << (vmi.pal_vm_info_2_s.rid_size -3 )) -1 ; -else { +if (max_ctx > (1U <<21 )) { + max_ctx = (1U <<21 ) -1 ;/* limit to 2^21 */ +printk (KERN_WARNING "cpu_init: max_ctx limited to 21 RID bits for bitmap size\n" ); + } + }else {printk (KERN_WARNING "cpu_init: PAL VM summary failed, assuming 18 RID bits\n" ); max_ctx = (1U <<15 ) -1 ;/* use architected minimum */ }
Hi Tony,
I have attached a patch for an issue with get_mmu_context() seen seen with RHEL 2.6.9-15 base code.
I would like you to take a look review and comment on it's merits. Bob Picco has review the patch prior to me senting it to you.
Data on problem:
Here is the results of modifing the code using a bitmap for looking up new context_id.
Updated png showing AIM7 shared run with before, after and clm data.
Lockstat Data:
There are 4 sets of lockstat data, one each for loads of 40K, 30K, 20K and 40K with no fork test. The lockstat data shows that as loading increases the lock contention on the task lock with wrap_mmu_context and higher utilization of the ia64_ctx lock and the ia64_global_tlb_purge lock.
Get_mmu_context() is called to get a new process context id number to uniquely identify it's address space and tbl entries. If the limit is reached the wrap_mmu_context is called to reset and flush the tlb's.
Wrap_mmu_context() uses ia64_ctx.max_ctx which is set based on ia64 arch. rid size minus 3 bits used for region number. For Itanium the size is 24 bits, so there are 21 bits used as an increment number. This number and the region register number (3bits) is used as the process address space number and is used in the tbl to identify tbl's associated with this process. So about 2^21 new processes can run before context_id wrapping occurs and the systems tbl's are flushed.
The difference in the number of times the ia64_ctx was acquired between running the fork test and not at 40K load is:
Lock Fork No Forkia64_ctx 9,111,760 1,162,555ia64_global_tlb_purge 101,371,392 33,888,711
Read Task Lock per second for wrap_mmu_context()
Load Locks/Sec Contention
20,000 5.8 20% 30,000 10.2 31% 40,000 14.6 39%
40,000 .0001 0% No Fork or Exec test
Notice the utilization percent for the ia64_ctx and tasklist_lock locks for the 4 runs.
20K 10% 30K 27% 40K 54%
40K 1% nofork
This utilization is based on the number of cycles the lock was busy for the measurement period.
Modified the number of bits used in the region id reg for context increment from 21 to 20 bits. This would cause wrap_mmu_context to be done sooner at 2^21.
Aim7 shared with fork:
Load 21bits 20bits
20K 66K 58K jobs/min 30K 58K 38K jobs/min 40K 44K 25K jobs/min
So now a load of 20K acts like 30K at 21bits and a load of 30K acts like 40K at 21bits.
The cost of having to search the entire task_list to find a free context number increases with number of processes in the tests. The following are the measure times for the wrap_mmu_context function.
Before:
Aim7 shared with fork/exec tests time spent in wrap_mmu_context() walking task loop.
Load Jobs/min Calls Maxprocs Cumulative Time Time/call Total Run
Time
30K 53279 33828 38198 864.0 sec 25.5 ms 62 min 40K 42378 79840 52496 2667.4 sec 33.4 ms 108 min 50K 31955 141000 75472 6011.5 sec 42.6 ms 180 min
Modified patch:
Aim7 shared with fork/exec tests time spent in wrap_mmu_context() using 128K bitmap.
Load Jobs/min Calls Maxprocs Cumulative Time Time/call Total Run
Time
30K 61350 1764278 na .312 sec 177 ns 48
min
40K 60858 3203092 na .561 sec 175 ns 64 min 50K 60826 5110677 na .887 sec 174 ns 80 in
Making the bitmap 512K reduced the tlb flush substantially but the jobs/min only increased by about 100-150. The 512k size is 1/4 of the max it could be, 2^21. The patch uses the total 2^21 size.
Lockstat data shows 0% contention of the rwlock task_list in wrap_mmu_context() and the utilization goes from 54% to 1.6%. The ia64_ctx lock utlization goes from 54% to 0.1% and the contention down to 0.3%.
Regards,
pete
Peter Keilty wrote on Thursday, October 27, 2005 10:28 AM
Quoted 9 lines
Please find attached IA64
context_idpatch and supporting data for your Review and consideration.
...
Lockstat Data:
There are 4 sets of lockstat data, one each for loads of 40K, 30K, 20K and 40K with no fork test. The lockstat data shows that as loading increases the lock contention on the task lock withwrap_mmu_contextand higher utilization of theia64_ctxlock and theia64_global_tlb_purgelock.
Current implementation in wrap_mmu_context did not fully utilize all the rid space at the time of wrap. It finds first available free range starting from ia64_ctx.next, presumably much smaller than max_ctx.
Was the lock contention because of much more frequent wrap_mmu_context? Ideally, it should only do one wrap when the entire rid space is exhausted. Current implementation in wrap_mmu_context is suboptimal in performance.
Quoted 7 lines
wrap_mmu_context(structmm_struct*mm)
{ ....arch/ia64/mm/tlb.c +0
-1@@ -52,28 +74,23 @@ ia64_ctx.limit = max_ctx +1 ;/* -* Scan all the task's mm->context and set proper safe range
arch/ia64/mm/tlb.c line 86
Quoted 10 lines
arch/ia64/mm/tlb.c +9
-0+* Scan the ia64_ctx bitmap and set proper safe range */ +repeat: + next_ctx =find_next_zero_bit (ia64_ctx.bitmap, ia64_ctx.limit, ia64_ctx.next); +if (next_ctx >= ia64_ctx.limit) { +smp_mb (); + ia64_ctx.next =300 ;/* skip daemons */ + goto repeat; + } + ia64_ctx.next = next_ctx;
I like the bitmap thing. But what's up with all this old range finding code doing here? You have a full bitmap that tracks used ctx_id, one more bitmap can be added to track pending flush. Then at the time of wrap, we can simply xor them to get full reusable rid. With that, kernel will only wrap when entire rid space is exhausted. I will post a patch.
- Ken
Chen, Kenneth W wrote on Thursday, October 27, 2005 7:55 PM
Quoted 6 lines
I like the bitmap thing. But what's up with all this old range finding code doing here? You have a full bitmap that tracks used
ctx_id, one more bitmap can be added to track pending flush. Then at the time of wrap, we can simply xor them to get full reusable rid. With that, kernel will only wrap when entire rid space is exhausted. I will post a patch.
Here is the patch, on top of Peter's patch: Add a flush bitmap to track which rid can be recycled when wrap happens. This optimization allows kernel to only wrap and flush tlb when the entire rid space is exhausted. It should dramatically reduce number of rid wrap frequency compare to current implementation.
Lightly tested, I will do more thorough testing. Also, I have a few things want to look at, especially in the area of setting the flushmap bit. There are a few other areas to fine tune Peter's original patch as well.
arch/ia64/mm/tlb.c renamed from arch/ia64/mm/tlb.c.orig +13 -23
@@ -4,10 +4,13 @@* Copyright (C) 1998-2001, 2003 Hewlett-Packard Co * David Mosberger-Tang <[email protected]> * +* Copyright (C) 2000, 2002-2003, 2005 Intel Co * 08/02/00 A. Mallick <[email protected]> * Modified RID allocation for SMP * Goutham Rao <[email protected]> * IPI based ptc implementation and A-step IPI implementation. +* Rohit Seth <[email protected]> +* Ken Chen <[email protected]> */ #include <linux/config.h> #include <linux/module.h>
@@ -33,7 +36,6 @@ static struct {struct ia64_ctx ia64_ctx = { .lock =SPIN_LOCK_UNLOCKED , .next =1 , - .limit = (1 <<15 ) -1 ,/* start out with the safe (architected) limit */ .max_ctx = ~0U };
@@ -55,6 +57,10 @@ mmu_context_init (void) (ia64_ctx.max_ctx+1 )>>3 ,PAGE_SIZE , __pa(MAX_DMA_ADDRESS)); + ia64_ctx.flushmap = (unsigned long *)__alloc_bootmem ( + (ia64_ctx.max_ctx+1 )>>3 , +PAGE_SIZE , +__pa (MAX_DMA_ADDRESS )); }spin_unlock_irqrestore (&ia64_ctx.lock, flags); }
@@ -65,30 +71,14 @@ mmu_context_init (void)void wrap_mmu_context (struct mm_struct *mm) { -unsigned int next_ctx, max_ctx = ia64_ctx.max_ctx;int i; -if (ia64_ctx.next > max_ctx) - ia64_ctx.next =300 ;/* skip daemons */ - ia64_ctx.limit = max_ctx +1 ; - -/* -* Scan the ia64_ctx bitmap and set proper safe range -*/ -repeat: - next_ctx =find_next_zero_bit (ia64_ctx.bitmap, ia64_ctx.limit, ia64_ctx.next); -if (next_ctx >= ia64_ctx.limit) { -smp_mb (); - ia64_ctx.next =300 ;/* skip daemons */ - goto repeat; - } - ia64_ctx.next = next_ctx; - - next_ctx =find_next_bit (ia64_ctx.bitmap, ia64_ctx.limit, ia64_ctx.next); -if (next_ctx >= ia64_ctx.limit) { - next_ctx = ia64_ctx.limit; - } - ia64_ctx.limit = next_ctx; +bitmap_xor (ia64_ctx.bitmap, ia64_ctx.bitmap, + ia64_ctx.flushmap, ia64_ctx.max_ctx); +bitmap_zero (ia64_ctx.flushmap, ia64_ctx.max_ctx); +/* use offset at 300 to skip daemons */ + ia64_ctx.next =find_next_zero_bit (ia64_ctx.bitmap, + ia64_ctx.max_ctx,300 );/* can't call flush_tlb_all() here because of race condition with O(1) scheduler [EF] */ {
include/asm-ia64/mmu_context.h renamed from include/asm-ia64/mmu_context.h.orig +5 -2
@@ -32,9 +32,10 @@struct ia64_ctx {spinlock_t lock;unsigned int next;/* next context number to use */ -unsigned int limit;/* next >= limit => must call wrap_mmu_context() */ unsigned int max_ctx;/* max. context value supported by all CPUs */ +/* next > max_ctx => must call wrap_mmu_context() */ unsigned long *bitmap;/* bitmap size is max_ctx+1 */ +unsigned long *flushmap;/* pending rid to be flushed */ };extern struct ia64_ctx ia64_ctx;
@@ -85,7 +86,9 @@ get_mmu_context (struct mm_struct *mm) context = mm->context;if (context =0 ) {cpus_clear (mm->cpu_vm_mask); -if (ia64_ctx.next >= ia64_ctx.limit) + ia64_ctx.next =find_next_zero_bit (ia64_ctx.bitmap, + ia64_ctx.max_ctx, ia64_ctx.next); +if (ia64_ctx.next >= ia64_ctx.max_ctx)wrap_mmu_context (mm); mm->context = context = ia64_ctx.next++;set_bit (context, ia64_ctx.bitmap);
include/asm-ia64/tlbflush.h renamed from include/asm-ia64/tlbflush.h.orig +2 -1
@@ -51,7 +51,8 @@ flush_tlb_mm (struct mm_struct *mm)if (!mm)return ; -clear_bit (mm->context, ia64_ctx.bitmap); +/* fix me: should we hold ia64_ctx.lock? */ +set_bit (mm->context, ia64_ctx.flushmap); mm->context =0 ;if (atomic_read (&mm->mm_users) =0 )
Peter Keilty wrote on Thursday, October 27, 2005 10:28 AM
Quoted 8 lines
Please find attached IA64
context_idpatch and supporting data for your Review and consideration.arch/ia64/kernel/setup.c +0
-0@@ -419,6 +419,7 @@
arch/ia64/kernel/setup.c line 808
Quoted 10 lines
arch/ia64/kernel/setup.c +6
-2/* set ia64_ctx.max_rid to the maximum RID that is supported by all CPUs: */ -if (ia64_pal_vm_summary (NULL , &vmi) =0 ) +if (ia64_pal_vm_summary (NULL , &vmi) =0 ) { max_ctx = (1U << (vmi.pal_vm_info_2_s.rid_size -3 )) -1 ; -else { +if (max_ctx > (1U <<21 )) { + max_ctx = (1U <<21 ) -1 ;/* limit to 2^21 */ +printk (KERN_WARNING "cpu_init: max_ctx limited to 21 RID bits for bitmap size\n" ); + } + }else {
This printk is spurious. When I first read it, my reaction is: rid as in region register is architecturally defined at maximum 24 bits. Printing 21-bit for rid is misleading. We currently use 8 rid number per process. So 21 bit for context number is the max. max_ctx will never exceed 1<<21. Suggest this printk to be dropped.
- Ken
Kenneth,
Quoted 30 lines
-----Original Message-----
From: Chen, Kenneth W [mailto:[email protected]]
Sent: Thursday, October 27, 2005 11:24 PM
To: 'Peter Keilty'; [email protected]
Cc: 'Eric Whitney'
Subject: RE: ia64get_mmu_contextpatchPeter Keilty wrote on Thursday, October 27, 2005 10:28 AM
Quoted 4 lines
Please find attached IA64
context_idpatch and supporting data for your Review and consideration.---
linux-2.6.14-rc3/arch/ia64/kernel/setup.c2005-10-26 10:54:06.000000000 -0400Quoted 1 line
+++
linux-2.6.14-rc3pmk/arch/ia64/kernel/setup.c2005-10-25 14:53:59.000000000 -0400Review
Quoted 4 lines
(quoted diff) +0
-0@@ -419,6 +419,7 @@ /* set ia64_ctx.max_rid to the maximum RID that issupported by all CPUs: */
Review
Quoted 3 lines
(quoted diff) +1
-1- if (ia64_pal_vm_summary(NULL, &vmi) = 0) + if (ia64_pal_vm_summary(NULL, &vmi) = 0) { max_ctx = (1U << (vmi.pal_vm_info_2_s.rid_size- 3)) - 1;
arch/ia64/kernel/setup.c
Quoted 3 lines
arch/ia64/kernel/setup.c +2
-1-else { +if (max_ctx > (1U <<21 )) { + max_ctx = (1U <<21 ) -1 ;/* limit to 2^21 */
Review
Quoted 1 line
(quoted diff) +1
-0+ printk(KERN_WARNING "cpu_init: max_ctxlimited to 21 RID bits for bitmap size\n");
Quoted 10 lines
arch/ia64/kernel/setup.c line 808
Quoted 2 lines
arch/ia64/kernel/setup.c +2
-0+ } + }else {This printk is spurious. When I first read it, my reaction
is: rid as in region register is architecturally defined at maximum 24 bits. Printing 21-bit for rid is misleading. We currently use 8 rid number per process.
So 21 bit for context number is the max.max_ctxwill never exceed 1<<21.
Suggest this printk to be dropped.
I add that per mail with Tony, He suggested that we might limit the bitmap size so that if the rid size was increase in future hardware that a Giga byte
Would not be allocated. The printk was to let people know that the rid was being
limited to 2^21 if the rid was enlarged.
Quoted 4 lines
- Ken
Hi Kenneth,
Quoted 17 lines
-----Original Message-----
From: Chen, Kenneth W [mailto:[email protected]]
Sent: Thursday, October 27, 2005 10:55 PM
To: 'Peter Keilty'; [email protected]
Cc: 'Eric Whitney'
Subject: RE: ia64get_mmu_contextpatchPeter Keilty wrote on Thursday, October 27, 2005 10:28 AM
Quoted 5 lines
Please find attached IA64
context_idpatch and supporting data for your Review and consideration.
...
Lockstat Data:
There are 4 sets of lockstat data, one each for loads of40K, 30K, 20K
Quoted 2 lines
and 40K with no fork test. The lockstat data shows that as loading increases the lock contention on the task lock with
wrap_mmu_context
Quoted 8 lines
Quoted 2 lines
and higher utilization of the
ia64_ctxlock and theia64_global_tlb_purgelock.Current implementation in
wrap_mmu_contextdid not fully utilize all the rid space at the time of wrap. It finds first available free range starting from ia64_ctx.next, presumably much smaller thanmax_ctx.
The original code did use full range, but once wrapping occurred yes ranging was used by setting limit. The ranging did go out to the the max_limit on follow on calls, but the range size could small Causing more calls to wrap_mmu_context.
Quoted 2 lines
Was the lock contention because of much more frequent
wrap_mmu_context?
Indirectly, The real reason was the time used to task_list walking derefencing pointers (tens of thousands of processes) trying to find a unused rid.
Quoted 16 lines
Ideally, it should only do one wrap when the entire rid space is exhausted. Current implementation in
wrap_mmu_contextis suboptimal in performance.arch/ia64/mm/tlb.c line 80
Quoted 10 lines
wrap_mmu_context(structmm_struct*mm) { ....arch/ia64/mm/tlb.c +3
-1@@ -52,28 +74,23 @@ ia64_ctx.limit = max_ctx +1 ;/* -* Scan all the task's mm->context and set proper safe range +* Scan the ia64_ctx bitmap and set proper safe range */ +repeat: + next_ctx = find_next_zero_bit(ia64_ctx.bitmap,ia64_ctx.limit, ia64_ctx.next);
Quoted 12 lines
arch/ia64/mm/tlb.c line 85
Quoted 6 lines
arch/ia64/mm/tlb.c +6
-0+if (next_ctx >= ia64_ctx.limit) { +smp_mb (); + ia64_ctx.next =300 ;/* skip daemons */ + goto repeat; + } + ia64_ctx.next = next_ctx;I like the bitmap thing. But what's up with all this old range finding code doing here? You have a full bitmap that tracks used
ctx_id, one more bitmap can be added to track pending flush. Then at the time of wrap, we can simply xor them to get full reusable rid.
I do like the flush bitmap idea. Although there is a cost call find_next_zero_bit
every time for a context_id instead of set up a range that is zero.
Quoted 1 line
With that, kernel will only wrap when entire rid space is exhausted.
Only from were the next index started from.
Quoted 5 lines
I will post a patch.
- Ken
Peter Keilty wrote on Friday, October 28, 2005 7:50 AM
Quoted 1 line
The original code did use full range,
Yes or no, first call to wrap_mmu_context occurs at ia64_ctx.next equals 2^15 (32768), then second call occurs at 2097152. First one is needlessly too early. One can say it uses full range. But the real thing I'm after is number of ctx_id allocation in between global tlb flush. Kernel did a global flush before entire 2M ctx_id is used. That is not a "full range" (as in use up all ctx_id before a global tlb flush).
Quoted 4 lines
but once wrapping
occurred yes ranging was used by setting limit. The ranging did go out to the themax_limiton follow on calls, but the range size could small Causing more calls towrap_mmu_context.
Exactly, ia64_ctx.next can only increment, while ia64_ctx.limit will move down. The code is effectively find_next_hole(), which isn't equivalent to find_largest_hole(). It could have a pathological worst case that you call wrap_mmu_context with only one ctx_id allocation. Worse, since next and limit pair can not cross a wrap around point, when next approach the end of the ctx_id space, the range it find is much smaller at that instance (though that should only occur once every 2M ctx id allocation).
Quoted 6 lines
Quoted 2 lines
Was the lock contention because of much more frequent
wrap_mmu_context?Indirectly, The real reason was the time used to
task_listwalking derefencing pointers (tens of thousands of processes) trying to find a unused rid.
What is the average number of ctx_id allocation between wrap_mmu_context call? That would tell us how efficient the current find_next_hole() is.
- Ken
Peter Keilty wrote on Friday, October 28, 2005 7:50 AM
Quoted 8 lines
Quoted 7 lines
This printk is spurious. When I first read it, my reaction
is: rid as in region register is architecturally defined at maximum 24 bits. Printing 21-bit for rid is misleading. We currently use 8 rid number per process.
So 21 bit for context number is the max.max_ctxwill never exceed 1<<21.
Suggest this printk to be dropped.
Quoted 5 lines
I add that per mail with Tony, He suggested that we might limit the bitmap size so that if the rid size was increase in future hardware that a Giga byte Would not be allocated. The printk was to let people know that the rid was being limited to 2^21 if the rid was enlarged.
If I were you, I would try to convince Tony with the argument I presented in my earlier post :-)
- Ken
Chen, Kenneth W wrote on Thursday, October 27, 2005 8:09 PM
Quoted 14 lines
Here is the patch, on top of Peter's patch: Add a flush bitmap to track which rid can be recycled when wrap happens. This optimization allows kernel to only wrap and flush tlb when the entire rid space is exhausted. It should dramatically reduce number of rid wrap frequency compare to current implementation.
Lightly tested, I will do more thorough testing. Also, I have a few things want to look at, especially in the area of setting the flushmap bit. There are a few other areas to fine tune Peter's original patch as well.
include/asm-ia64/tlbflush.h flush_tlb_mm (struct mm_struct *mm) · line 56
Quoted 7 lines
include/asm-ia64/tlbflush.h renamed from include/asm-ia64/tlbflush.h.orig +2
-1@@ -51,7 +51,8 @@ flush_tlb_mm (struct mm_struct *mm)if (!mm)return ; -clear_bit (mm->context, ia64_ctx.bitmap); +/* fix me: should we hold ia64_ctx.lock? */ +set_bit (mm->context, ia64_ctx.flushmap); mm->context =0 ;
I convinced myself that I really need a lock here. Otherwise, It will have write-after-write race with flushmap and cause ctx_id to leak. Spin lock is the wrong kind of lock to use here because it should allow concurrent updates between wrap_mmu_context().
arch/ia64/mm/tlb.c renamed from arch/ia64/mm/tlb.c.orig +4 -0
@@ -35,6 +35,7 @@ static struct {struct ia64_ctx ia64_ctx = { .lock =SPIN_LOCK_UNLOCKED , + .flushmap_rwlock =RW_LOCK_UNLOCKED , .next =1 , .max_ctx = ~0U };
@@ -73,9 +74,12 @@ wrap_mmu_context (struct mm_struct *mm) {int i; +write_lock (&ia64_ctx.flushmap_rwlock);bitmap_xor (ia64_ctx.bitmap, ia64_ctx.bitmap, ia64_ctx.flushmap, ia64_ctx.max_ctx);bitmap_zero (ia64_ctx.flushmap, ia64_ctx.max_ctx); +write_unlock (&ia64_ctx.flushmap_rwlock); +/* use offset at 300 to skip daemons */ ia64_ctx.next =find_next_zero_bit (ia64_ctx.bitmap, ia64_ctx.max_ctx,300 );
include/asm-ia64/mmu_context.h renamed from include/asm-ia64/mmu_context.h.orig +2 -1
@@ -31,11 +31,12 @@struct ia64_ctx {spinlock_t lock; +rwlock_t flushmap_rwlock;unsigned int next;/* next context number to use */ unsigned int max_ctx;/* max. context value supported by all CPUs */ /* next > max_ctx => must call wrap_mmu_context() */ unsigned long *bitmap;/* bitmap size is max_ctx+1 */ -unsigned long *flushmap;/* pending rid to be flushed */ +unsigned long *flushmap;/* pending ctx id to be flushed */ };extern struct ia64_ctx ia64_ctx;
include/asm-ia64/tlbflush.h renamed from include/asm-ia64/tlbflush.h.orig +3 -2
@@ -52,8 +52,9 @@ flush_tlb_mm (struct mm_struct *mm)if (!mm)return ; -/* fix me: should we hold ia64_ctx.lock? */ -set_bit (mm->context, ia64_ctx.flushmap); +read_lock (&ia64_ctx.flushmap_rwlock); +set_bit (mm->context, ia64_ctx.flushmap); +read_unlock (&ia64_ctx.flushmap_rwlock); mm->context =0 ;if (atomic_read (&mm->mm_users) =0 )
I did a crude fork bomb test, then printed out ia64_ctx.next ia64_ctx.limit and the diff in wrap_mmu_context(). Here is the result:
First call resulted expected 2M hole. Look what happens after first wrap. My worst fear of global tlb flush after just one ctx_id allocation is actually happening! Average ~ 90,000 ctx_id allocation between flush. Much smaller than advertised 2^21. This find_next_hole() is horribly broken in terms of performance.
- Ken
wrap_mmu_context: next = 32767, limit = 2097152 n = 2064385wrap_mmu_context: next = 300, limit = 1782 n = 1482wrap_mmu_context: next = 1783, limit = 1872 n = 89wrap_mmu_context: next = 1873, limit = 1883 n = 10wrap_mmu_context: next = 1884, limit = 1921 n = 37wrap_mmu_context: next = 1922, limit = 1973 n = 51wrap_mmu_context: next = 1974, limit = 1998 n = 24wrap_mmu_context: next = 1999, limit = 2055 n = 56wrap_mmu_context: next = 2056, limit = 2109 n = 53wrap_mmu_context: next = 2110, limit = 2111 n = 1wrap_mmu_context: next = 2118, limit = 2124 n = 6wrap_mmu_context: next = 2125, limit = 2128 n = 3wrap_mmu_context: next = 2129, limit = 140411 n = 138282wrap_mmu_context: next = 140412, limit = 518442 n = 378030wrap_mmu_context: next = 518443, limit = 520789 n = 2346wrap_mmu_context: next = 520790, limit = 980204 n = 459414wrap_mmu_context: next = 980205, limit = 1864365 n = 884160wrap_mmu_context: next = 1864366, limit = 1864528 n = 162wrap_mmu_context: next = 1864529, limit = 2066917 n = 202388wrap_mmu_context: next = 2066918, limit = 2097152 n = 30234wrap_mmu_context: next = 300, limit = 1782 n = 1482wrap_mmu_context: next = 1783, limit = 1872 n = 89wrap_mmu_context: next = 1873, limit = 1883 n = 10wrap_mmu_context: next = 1884, limit = 1921 n = 37wrap_mmu_context: next = 1922, limit = 1973 n = 51wrap_mmu_context: next = 1974, limit = 1998 n = 24wrap_mmu_context: next = 1999, limit = 2055 n = 56wrap_mmu_context: next = 2056, limit = 2109 n = 53wrap_mmu_context: next = 2110, limit = 2111 n = 1wrap_mmu_context: next = 2118, limit = 2124 n = 6wrap_mmu_context: next = 2125, limit = 2128 n = 3wrap_mmu_context: next = 2129, limit = 140411 n = 138282wrap_mmu_context: next = 140412, limit = 184686 n = 44274wrap_mmu_context: next = 184687, limit = 188664 n = 3977wrap_mmu_context: next = 188665, limit = 398496 n = 209831wrap_mmu_context: next = 398497, limit = 492604 n = 94107wrap_mmu_context: next = 492605, limit = 510747 n = 18142wrap_mmu_context: next = 510748, limit = 510923 n = 175wrap_mmu_context: next = 510924, limit = 518442 n = 7518wrap_mmu_context: next = 518443, limit = 520789 n = 2346wrap_mmu_context: next = 520790, limit = 980204 n = 459414wrap_mmu_context: next = 980205, limit = 1864365 n = 884160wrap_mmu_context: next = 1864366, limit = 1864528 n = 162wrap_mmu_context: next = 1864529, limit = 2097152 n = 232623wrap_mmu_context: next = 300, limit = 1782 n = 1482wrap_mmu_context: next = 1783, limit = 1872 n = 89wrap_mmu_context: next = 1873, limit = 1883 n = 10wrap_mmu_context: next = 1884, limit = 1921 n = 37wrap_mmu_context: next = 1922, limit = 1973 n = 51wrap_mmu_context: next = 1974, limit = 1998 n = 24wrap_mmu_context: next = 1999, limit = 2055 n = 56wrap_mmu_context: next = 2056, limit = 2109 n = 53wrap_mmu_context: next = 2110, limit = 2111 n = 1wrap_mmu_context: next = 2118, limit = 2124 n = 6wrap_mmu_context: next = 2125, limit = 2128 n = 3wrap_mmu_context: next = 2129, limit = 140411 n = 138282wrap_mmu_context: next = 140412, limit = 184686 n = 44274wrap_mmu_context: next = 184687, limit = 188664 n = 3977wrap_mmu_context: next = 188665, limit = 398496 n = 209831wrap_mmu_context: next = 398497, limit = 492604 n = 94107wrap_mmu_context: next = 492605, limit = 510747 n = 18142wrap_mmu_context: next = 510748, limit = 510923 n = 175wrap_mmu_context: next = 510924, limit = 518442 n = 7518wrap_mmu_context: next = 518443, limit = 520789 n = 2346wrap_mmu_context: next = 520790, limit = 980204 n = 459414wrap_mmu_context: next = 980205, limit = 1864365 n = 884160wrap_mmu_context: next = 1864366, limit = 1864528 n = 162wrap_mmu_context: next = 1864529, limit = 2097152 n = 232623
Ken,
It well be worth doing find_next_zero_bit from get_mmu_context Per your patch instead of the global tbl flush.
Quoted 91 lines
-----Original Message-----
From: Chen, Kenneth W [mailto:[email protected]]
Sent: Friday, October 28, 2005 2:41 PM
To: 'Peter Keilty'; [email protected]
Cc: 'Eric Whitney'
Subject: RE: ia64get_mmu_contextpatchI did a crude fork bomb test, then printed out ia64_ctx.next ia64_ctx.limit and the diff in
wrap_mmu_context(). Here is the result:First call resulted expected 2M hole. Look what happens after first wrap. My worst fear of global tlb flush after just one
ctx_idallocation is actually happening! Average ~ 90,000ctx_idallocation between flush. Much smaller than advertised 2^21. Thisfind_next_hole()is horribly broken in terms of performance.- Ken
wrap_mmu_context: next = 32767, limit = 2097152 n = 2064385wrap_mmu_context: next = 300, limit = 1782 n = 1482wrap_mmu_context: next = 1783, limit = 1872 n = 89wrap_mmu_context: next = 1873, limit = 1883 n = 10wrap_mmu_context: next = 1884, limit = 1921 n = 37wrap_mmu_context: next = 1922, limit = 1973 n = 51wrap_mmu_context: next = 1974, limit = 1998 n = 24wrap_mmu_context: next = 1999, limit = 2055 n = 56wrap_mmu_context: next = 2056, limit = 2109 n = 53wrap_mmu_context: next = 2110, limit = 2111 n = 1wrap_mmu_context: next = 2118, limit = 2124 n = 6wrap_mmu_context: next = 2125, limit = 2128 n = 3wrap_mmu_context: next = 2129, limit = 140411 n = 138282wrap_mmu_context: next = 140412, limit = 518442 n = 378030wrap_mmu_context: next = 518443, limit = 520789 n = 2346wrap_mmu_context: next = 520790, limit = 980204 n = 459414wrap_mmu_context: next = 980205, limit = 1864365 n = 884160wrap_mmu_context: next = 1864366, limit = 1864528 n = 162wrap_mmu_context: next = 1864529, limit = 2066917 n = 202388wrap_mmu_context: next = 2066918, limit = 2097152 n = 30234wrap_mmu_context: next = 300, limit = 1782 n = 1482wrap_mmu_context: next = 1783, limit = 1872 n = 89wrap_mmu_context: next = 1873, limit = 1883 n = 10wrap_mmu_context: next = 1884, limit = 1921 n = 37wrap_mmu_context: next = 1922, limit = 1973 n = 51wrap_mmu_context: next = 1974, limit = 1998 n = 24wrap_mmu_context: next = 1999, limit = 2055 n = 56wrap_mmu_context: next = 2056, limit = 2109 n = 53wrap_mmu_context: next = 2110, limit = 2111 n = 1wrap_mmu_context: next = 2118, limit = 2124 n = 6wrap_mmu_context: next = 2125, limit = 2128 n = 3wrap_mmu_context: next = 2129, limit = 140411 n = 138282wrap_mmu_context: next = 140412, limit = 184686 n = 44274wrap_mmu_context: next = 184687, limit = 188664 n = 3977wrap_mmu_context: next = 188665, limit = 398496 n = 209831wrap_mmu_context: next = 398497, limit = 492604 n = 94107wrap_mmu_context: next = 492605, limit = 510747 n = 18142wrap_mmu_context: next = 510748, limit = 510923 n = 175wrap_mmu_context: next = 510924, limit = 518442 n = 7518wrap_mmu_context: next = 518443, limit = 520789 n = 2346wrap_mmu_context: next = 520790, limit = 980204 n = 459414wrap_mmu_context: next = 980205, limit = 1864365 n = 884160wrap_mmu_context: next = 1864366, limit = 1864528 n = 162wrap_mmu_context: next = 1864529, limit = 2097152 n = 232623wrap_mmu_context: next = 300, limit = 1782 n = 1482wrap_mmu_context: next = 1783, limit = 1872 n = 89wrap_mmu_context: next = 1873, limit = 1883 n = 10wrap_mmu_context: next = 1884, limit = 1921 n = 37wrap_mmu_context: next = 1922, limit = 1973 n = 51wrap_mmu_context: next = 1974, limit = 1998 n = 24wrap_mmu_context: next = 1999, limit = 2055 n = 56wrap_mmu_context: next = 2056, limit = 2109 n = 53wrap_mmu_context: next = 2110, limit = 2111 n = 1wrap_mmu_context: next = 2118, limit = 2124 n = 6wrap_mmu_context: next = 2125, limit = 2128 n = 3wrap_mmu_context: next = 2129, limit = 140411 n = 138282wrap_mmu_context: next = 140412, limit = 184686 n = 44274wrap_mmu_context: next = 184687, limit = 188664 n = 3977wrap_mmu_context: next = 188665, limit = 398496 n = 209831wrap_mmu_context: next = 398497, limit = 492604 n = 94107wrap_mmu_context: next = 492605, limit = 510747 n = 18142wrap_mmu_context: next = 510748, limit = 510923 n = 175wrap_mmu_context: next = 510924, limit = 518442 n = 7518wrap_mmu_context: next = 518443, limit = 520789 n = 2346wrap_mmu_context: next = 520790, limit = 980204 n = 459414wrap_mmu_context: next = 980205, limit = 1864365 n = 884160wrap_mmu_context: next = 1864366, limit = 1864528 n = 162wrap_mmu_context: next = 1864529, limit = 2097152 n = 232623