Skip to contentSkip to conversation

ia64 get_mmu_context patch

PATCH Peter Keilty 2 participants 11 messages Oct 27, 2005 – Oct 28, 2005

lore
Share & tools
lore
#0 lore raw

Gentleman,

Please find attached IA64 context_id patch and supporting data for your Review and consideration.

Regards,
pete

Signed-off-by:

include/asm-ia64/mmu_context.h +3 -0
@@ -34,11 +34,13 @@
 	unsigned int next;	/* next context number to use */
 	unsigned int limit;	/* next >= limit => must call wrap_mmu_context() */
 	unsigned int max_ctx;	/* max. context value supported by all CPUs */
+	unsigned long *bitmap;	/* bitmap size is max_ctx+1 */
 };
 
 extern struct ia64_ctx ia64_ctx;
 DECLARE_PER_CPU(u8, ia64_need_tlb_flush);
 
+extern void mmu_context_init (void);
 extern void wrap_mmu_context (struct mm_struct *mm);
 
 static inline void
@@ -86,6 +88,7 @@
 				if (ia64_ctx.next >= ia64_ctx.limit)
 					wrap_mmu_context(mm);
 				mm->context = context = ia64_ctx.next++;
+	 			set_bit(context, ia64_ctx.bitmap);
 			}
 		}
 		spin_unlock_irqrestore(&ia64_ctx.lock, flags);
include/asm-ia64/tlbflush.h +1 -0
@@ -51,6 +51,7 @@
 	if (!mm)
 		return;
 
+	clear_bit(mm->context, ia64_ctx.bitmap);
 	mm->context = 0;
 
 	if (atomic_read(&mm->mm_users) == 0)
arch/ia64/mm/tlb.c +37 -21

90 lines not on this page · raw message

arch/ia64/kernel/setup.c +7 -2
@@ -419,6 +419,7 @@
 #endif
 
 	cpu_init();	/* initialize the bootstrap CPU */
+	mmu_context_init();	/* initialize context_id bitmap */
 
 #ifdef CONFIG_ACPI
 	acpi_boot_init();
@@ -798,9 +799,13 @@
 #endif
 
 	/* set ia64_ctx.max_rid to the maximum RID that is supported by all CPUs: */
-	if (ia64_pal_vm_summary(NULL, &vmi) == 0)
+	if (ia64_pal_vm_summary(NULL, &vmi) == 0) {
 		max_ctx = (1U << (vmi.pal_vm_info_2_s.rid_size - 3)) - 1;
-	else {
+		if (max_ctx > (1U << 21)) {
+			max_ctx = (1U << 21) - 1;	/* limit to 2^21  */
+			printk(KERN_WARNING "cpu_init: max_ctx limited to 21 RID bits for bitmap size\n");
+		}
+	} else {
 		printk(KERN_WARNING "cpu_init: PAL VM summary failed, assuming 18 RID bits\n");
 		max_ctx = (1U << 15) - 1;	/* use architected minimum */
 	}

Hi Tony,

I have attached a patch for an issue with get_mmu_context() seen seen with RHEL 2.6.9-15 base code.
I would like you to take a look review and comment on it's merits. Bob Picco has review the patch prior to me senting it to you.

Data on problem:

Here is the results of modifing the code using a bitmap for looking up new context_id.

Updated png showing AIM7 shared run with before, after and clm data.

Lockstat Data:
There are 4 sets of lockstat data, one each for loads of 40K, 30K, 20K and 40K with no fork test. The lockstat data shows that as loading increases the lock contention on the task lock with wrap_mmu_context and higher utilization of the ia64_ctx lock and the ia64_global_tlb_purge lock.

Get_mmu_context() is called to get a new process context id number to uniquely identify it's address space and tbl entries. If the limit is reached the wrap_mmu_context is called to reset and flush the tlb's.

Wrap_mmu_context() uses ia64_ctx.max_ctx which is set based on ia64 arch. rid size minus 3 bits used for region number. For Itanium the size is 24 bits, so there are 21 bits used as an increment number. This number and the region register number (3bits) is used as the process address space number and is used in the tbl to identify tbl's associated with this process. So about 2^21 new processes can run before context_id wrapping occurs and the systems tbl's are flushed.

The difference in the number of times the ia64_ctx was acquired between running the fork test and not at 40K load is:
Lock Fork No Fork
ia64_ctx 9,111,760 1,162,555
ia64_global_tlb_purge 101,371,392 33,888,711

Read Task Lock per second for wrap_mmu_context()
Load Locks/Sec Contention

20,000         5.8           20%
30,000        10.2           31%
40,000        14.6           39%

40,000 .0001 0% No Fork or Exec test

Notice the utilization percent for the ia64_ctx and tasklist_lock locks for the 4 runs.

20K    10%
30K    27%
40K    54%

40K 1% nofork
This utilization is based on the number of cycles the lock was busy for the measurement period.

Modified the number of bits used in the region id reg for context increment from 21 to 20 bits. This would cause wrap_mmu_context to be done sooner at 2^21.

Aim7 shared with fork:
Load 21bits 20bits

20K    66K      58K  jobs/min
30K    58K      38K  jobs/min
40K    44K      25K  jobs/min

So now a load of 20K acts like 30K at 21bits and a load of 30K acts like 40K at 21bits.

The cost of having to search the entire task_list to find a free context number increases with number of processes in the tests. The following are the measure times for the wrap_mmu_context function.

Before:
Aim7 shared with fork/exec tests time spent in wrap_mmu_context() walking task loop.
Load Jobs/min Calls Maxprocs Cumulative Time Time/call Total Run
Time

30K      53279    33828     38198        864.0 sec    25.5 ms     62 min   
40K      42378    79840     52496       2667.4 sec    33.4 ms    108 min
50K      31955   141000     75472       6011.5 sec    42.6 ms    180 min

Modified patch:
Aim7 shared with fork/exec tests time spent in wrap_mmu_context() using 128K bitmap.
Load Jobs/min Calls Maxprocs Cumulative Time Time/call Total Run
Time
30K 61350 1764278 na .312 sec 177 ns 48
min

40K	   60858   3203092     na	     .561 sec	 175 ns      64 min
50K	   60826   5110677     na	     .887 sec      174 ns      80 in

Making the bitmap 512K reduced the tlb flush substantially but the jobs/min only increased by about 100-150. The 512k size is 1/4 of the max it could be, 2^21. The patch uses the total 2^21 size.

Lockstat data shows 0% contention of the rwlock task_list in wrap_mmu_context() and the utilization goes from 54% to 1.6%. The ia64_ctx lock utlization goes from 54% to 0.1% and the contention down to 0.3%.

Regards,
pete

#1 lore raw

Peter Keilty wrote on Thursday, October 27, 2005 10:28 AM

Quoted 9 lines

Please find attached IA64 context_id patch and supporting data for your Review and consideration.
...
Lockstat Data:
There are 4 sets of lockstat data, one each for loads of 40K, 30K, 20K and 40K with no fork test. The lockstat data shows that as loading increases the lock contention on the task lock with wrap_mmu_context and higher utilization of the ia64_ctx lock and the ia64_global_tlb_purge lock.

Current implementation in wrap_mmu_context did not fully utilize all the rid space at the time of wrap. It finds first available free range starting from ia64_ctx.next, presumably much smaller than max_ctx.

Was the lock contention because of much more frequent wrap_mmu_context? Ideally, it should only do one wrap when the entire rid space is exhausted. Current implementation in wrap_mmu_context is suboptimal in performance.

Quoted 7 lines

wrap_mmu_context (struct mm_struct *mm)
{ ....

arch/ia64/mm/tlb.c +0 -1
@@ -52,28 +74,23 @@
 	ia64_ctx.limit = max_ctx + 1;
 
 	/*
-	 * Scan all the task's mm->context and set proper safe range

arch/ia64/mm/tlb.c line 86

Quoted 10 lines
arch/ia64/mm/tlb.c +9 -0
+	 * Scan the ia64_ctx bitmap and set proper safe range
 	 */
+repeat:
+	next_ctx = find_next_zero_bit(ia64_ctx.bitmap, ia64_ctx.limit, ia64_ctx.next);
+	if (next_ctx >= ia64_ctx.limit) {
+		smp_mb();
+		ia64_ctx.next = 300;	/* skip daemons */
+		goto repeat;
+	}
+	ia64_ctx.next = next_ctx;

I like the bitmap thing. But what's up with all this old range finding code doing here? You have a full bitmap that tracks used ctx_id, one more bitmap can be added to track pending flush. Then at the time of wrap, we can simply xor them to get full reusable rid. With that, kernel will only wrap when entire rid space is exhausted. I will post a patch.

- Ken

#2 lore raw
In reply to Peter Keilty

Chen, Kenneth W wrote on Thursday, October 27, 2005 7:55 PM

Quoted 6 lines

I like the bitmap thing. But what's up with all this old range finding code doing here? You have a full bitmap that tracks used ctx_id, one more bitmap can be added to track pending flush. Then at the time of wrap, we can simply xor them to get full reusable rid. With that, kernel will only wrap when entire rid space is exhausted. I will post a patch.

Here is the patch, on top of Peter's patch: Add a flush bitmap to track which rid can be recycled when wrap happens. This optimization allows kernel to only wrap and flush tlb when the entire rid space is exhausted. It should dramatically reduce number of rid wrap frequency compare to current implementation.

Lightly tested, I will do more thorough testing. Also, I have a few things want to look at, especially in the area of setting the flushmap bit. There are a few other areas to fine tune Peter's original patch as well.

arch/ia64/mm/tlb.c renamed from arch/ia64/mm/tlb.c.orig +13 -23
@@ -4,10 +4,13 @@
  * Copyright (C) 1998-2001, 2003 Hewlett-Packard Co
  *	David Mosberger-Tang <[email protected]>
  *
+ * Copyright (C) 2000, 2002-2003, 2005 Intel Co
  * 08/02/00 A. Mallick <[email protected]>
  *		Modified RID allocation for SMP
  *          Goutham Rao <[email protected]>
  *              IPI based ptc implementation and A-step IPI implementation.
+ * Rohit Seth <[email protected]>
+ * Ken Chen <[email protected]>
  */
 #include <linux/config.h>
 #include <linux/module.h>
@@ -33,7 +36,6 @@ static struct {
 struct ia64_ctx ia64_ctx = {
 	.lock =		SPIN_LOCK_UNLOCKED,
 	.next =		1,
-	.limit =	(1 << 15) - 1,		/* start out with the safe (architected) limit */
 	.max_ctx =	~0U
 };
 
@@ -55,6 +57,10 @@ mmu_context_init (void)
 							(ia64_ctx.max_ctx+1)>>3,
 							PAGE_SIZE,
 							__pa(MAX_DMA_ADDRESS));
+		ia64_ctx.flushmap = (unsigned long *)__alloc_bootmem(
+							(ia64_ctx.max_ctx+1)>>3,
+							PAGE_SIZE,
+							__pa(MAX_DMA_ADDRESS));
 	}
 	spin_unlock_irqrestore(&ia64_ctx.lock, flags);
 }
@@ -65,30 +71,14 @@ mmu_context_init (void)
 void
 wrap_mmu_context (struct mm_struct *mm)
 {
-	unsigned int next_ctx, max_ctx = ia64_ctx.max_ctx;
 	int i;
 
-	if (ia64_ctx.next > max_ctx)
-		ia64_ctx.next = 300;	/* skip daemons */
-	ia64_ctx.limit = max_ctx + 1;
-
-	/*
-	 * Scan the ia64_ctx bitmap and set proper safe range
-	 */
-repeat:
-	next_ctx = find_next_zero_bit(ia64_ctx.bitmap, ia64_ctx.limit, ia64_ctx.next);
-	if (next_ctx >= ia64_ctx.limit) {
-		smp_mb();
-		ia64_ctx.next = 300;	/* skip daemons */
-		goto repeat;
-	}
-	ia64_ctx.next = next_ctx;
-
-	next_ctx = find_next_bit(ia64_ctx.bitmap, ia64_ctx.limit, ia64_ctx.next);
-	if (next_ctx >= ia64_ctx.limit) {
-		next_ctx = ia64_ctx.limit;
-	}
-	ia64_ctx.limit = next_ctx;
+	bitmap_xor(ia64_ctx.bitmap, ia64_ctx.bitmap,
+		   ia64_ctx.flushmap, ia64_ctx.max_ctx);
+	bitmap_zero(ia64_ctx.flushmap, ia64_ctx.max_ctx);
+	/* use offset at 300 to skip daemons */
+	ia64_ctx.next = find_next_zero_bit(ia64_ctx.bitmap,
+				ia64_ctx.max_ctx, 300);
 
 	/* can't call flush_tlb_all() here because of race condition with O(1) scheduler [EF] */
 	{
include/asm-ia64/mmu_context.h renamed from include/asm-ia64/mmu_context.h.orig +5 -2
@@ -32,9 +32,10 @@
 struct ia64_ctx {
 	spinlock_t lock;
 	unsigned int next;	/* next context number to use */
-	unsigned int limit;	/* next >= limit => must call wrap_mmu_context() */
 	unsigned int max_ctx;	/* max. context value supported by all CPUs */
+				/* next > max_ctx => must call wrap_mmu_context() */
 	unsigned long *bitmap;	/* bitmap size is max_ctx+1 */
+	unsigned long *flushmap;/* pending rid to be flushed */
 };
 
 extern struct ia64_ctx ia64_ctx;
@@ -85,7 +86,9 @@ get_mmu_context (struct mm_struct *mm)
 			context = mm->context;
 			if (context = 0) {
 				cpus_clear(mm->cpu_vm_mask);
-				if (ia64_ctx.next >= ia64_ctx.limit)
+				ia64_ctx.next = find_next_zero_bit(ia64_ctx.bitmap,
+						ia64_ctx.max_ctx, ia64_ctx.next);
+				if (ia64_ctx.next >= ia64_ctx.max_ctx)
 					wrap_mmu_context(mm);
 				mm->context = context = ia64_ctx.next++;
 	 			set_bit(context, ia64_ctx.bitmap);
include/asm-ia64/tlbflush.h renamed from include/asm-ia64/tlbflush.h.orig +2 -1
@@ -51,7 +51,8 @@ flush_tlb_mm (struct mm_struct *mm)
 	if (!mm)
 		return;
 
-	clear_bit(mm->context, ia64_ctx.bitmap);
+	/* fix me: should we hold ia64_ctx.lock? */
+	set_bit(mm->context, ia64_ctx.flushmap);
 	mm->context = 0;
 
 	if (atomic_read(&mm->mm_users) = 0)
#3 lore raw
In reply to Peter Keilty

Peter Keilty wrote on Thursday, October 27, 2005 10:28 AM

Quoted 8 lines

Please find attached IA64 context_id patch and supporting data for your Review and consideration.

arch/ia64/kernel/setup.c +0 -0
@@ -419,6 +419,7 @@
...
 

arch/ia64/kernel/setup.c line 808

Quoted 10 lines
arch/ia64/kernel/setup.c +6 -2
 	/* set ia64_ctx.max_rid to the maximum RID that is supported by all CPUs: */
-	if (ia64_pal_vm_summary(NULL, &vmi) = 0)
+	if (ia64_pal_vm_summary(NULL, &vmi) = 0) {
 		max_ctx = (1U << (vmi.pal_vm_info_2_s.rid_size - 3)) - 1;
-	else {
+		if (max_ctx > (1U << 21)) {
+			max_ctx = (1U << 21) - 1;	/* limit to 2^21  */
+			printk(KERN_WARNING "cpu_init: max_ctx limited to 21 RID bits for bitmap size\n");
+		}
+	} else {

This printk is spurious. When I first read it, my reaction is: rid as in region register is architecturally defined at maximum 24 bits. Printing 21-bit for rid is misleading. We currently use 8 rid number per process. So 21 bit for context number is the max. max_ctx will never exceed 1<<21. Suggest this printk to be dropped.

- Ken

#4 lore raw
In reply to Peter Keilty

Kenneth,

Quoted 30 lines

-----Original Message-----
From: Chen, Kenneth W [mailto:[email protected]]
Sent: Thursday, October 27, 2005 11:24 PM
To: 'Peter Keilty'; [email protected]
Cc: 'Eric Whitney'
Subject: RE: ia64 get_mmu_context patch

Peter Keilty wrote on Thursday, October 27, 2005 10:28 AM

Quoted 4 lines

Please find attached IA64 context_id patch and supporting data for your Review and consideration.

--- linux-2.6.14-rc3/arch/ia64/kernel/setup.c

2005-10-26 10:54:06.000000000 -0400
Quoted 1 line

+++ linux-2.6.14-rc3pmk/arch/ia64/kernel/setup.c

2005-10-25 14:53:59.000000000 -0400

Review

Quoted 4 lines
(quoted diff) +0 -0
@@ -419,6 +419,7 @@
...
 
 	/* set ia64_ctx.max_rid to the maximum RID that is 

supported by all CPUs: */

Review

Quoted 3 lines
(quoted diff) +1 -1
-	if (ia64_pal_vm_summary(NULL, &vmi) = 0)
+	if (ia64_pal_vm_summary(NULL, &vmi) = 0) {
 		max_ctx = (1U << (vmi.pal_vm_info_2_s.rid_size 

- 3)) - 1;

arch/ia64/kernel/setup.c

Quoted 3 lines
arch/ia64/kernel/setup.c +2 -1
-	else {
+		if (max_ctx > (1U << 21)) {
+			max_ctx = (1U << 21) - 1;	/* 

limit to 2^21 */

Review

Quoted 1 line
(quoted diff) +1 -0
+			printk(KERN_WARNING "cpu_init: max_ctx 

limited to 21 RID bits for bitmap size\n");

Quoted 10 lines

arch/ia64/kernel/setup.c line 808

Quoted 2 lines
arch/ia64/kernel/setup.c +2 -0
+		}
+	} else {

This printk is spurious. When I first read it, my reaction
is: rid as in region register is architecturally defined at maximum 24 bits. Printing 21-bit for rid is misleading. We currently use 8 rid number per process.
So 21 bit for context number is the max. max_ctx will never exceed 1<<21.
Suggest this printk to be dropped.

I add that per mail with Tony, He suggested that we might limit the bitmap size so that if the rid size was increase in future hardware that a Giga byte
Would not be allocated. The printk was to let people know that the rid was being
limited to 2^21 if the rid was enlarged.

Quoted 4 lines

- Ken

#5 lore raw
In reply to Peter Keilty

Hi Kenneth,

Quoted 17 lines

-----Original Message-----
From: Chen, Kenneth W [mailto:[email protected]]
Sent: Thursday, October 27, 2005 10:55 PM
To: 'Peter Keilty'; [email protected]
Cc: 'Eric Whitney'
Subject: RE: ia64 get_mmu_context patch

Peter Keilty wrote on Thursday, October 27, 2005 10:28 AM

Quoted 5 lines

Please find attached IA64 context_id patch and supporting data for your Review and consideration.
...
Lockstat Data:
There are 4 sets of lockstat data, one each for loads of

40K, 30K, 20K

Quoted 2 lines

and 40K with no fork test. The lockstat data shows that as loading increases the lock contention on the task lock with

wrap_mmu_context

Quoted 8 lines
Quoted 2 lines

and higher utilization of the ia64_ctx lock and the ia64_global_tlb_purge lock.

Current implementation in wrap_mmu_context did not fully utilize all the rid space at the time of wrap. It finds first available free range starting from ia64_ctx.next, presumably much smaller than max_ctx.

The original code did use full range, but once wrapping occurred yes ranging was used by setting limit. The ranging did go out to the the max_limit on follow on calls, but the range size could small Causing more calls to wrap_mmu_context.

Quoted 2 lines

Was the lock contention because of much more frequent wrap_mmu_context?

Indirectly, The real reason was the time used to task_list walking derefencing pointers (tens of thousands of processes) trying to find a unused rid.

Quoted 16 lines

Ideally, it should only do one wrap when the entire rid space is exhausted. Current implementation in wrap_mmu_context is suboptimal in performance.

arch/ia64/mm/tlb.c line 80

Quoted 10 lines

wrap_mmu_context (struct mm_struct *mm) { ....

arch/ia64/mm/tlb.c +3 -1
@@ -52,28 +74,23 @@
 	ia64_ctx.limit = max_ctx + 1;
 
 	/*
-	 * Scan all the task's mm->context and set proper safe range
+	 * Scan the ia64_ctx bitmap and set proper safe range
 	 */
+repeat:
+	next_ctx = find_next_zero_bit(ia64_ctx.bitmap, 

ia64_ctx.limit, ia64_ctx.next);

Quoted 12 lines

arch/ia64/mm/tlb.c line 85

Quoted 6 lines
arch/ia64/mm/tlb.c +6 -0
+	if (next_ctx >= ia64_ctx.limit) {
+		smp_mb();
+		ia64_ctx.next = 300;	/* skip daemons */
+		goto repeat;
+	}
+	ia64_ctx.next = next_ctx;

I like the bitmap thing. But what's up with all this old range finding code doing here? You have a full bitmap that tracks used ctx_id, one more bitmap can be added to track pending flush. Then at the time of wrap, we can simply xor them to get full reusable rid.

I do like the flush bitmap idea. Although there is a cost call find_next_zero_bit
every time for a context_id instead of set up a range that is zero.

Quoted 1 line

With that, kernel will only wrap when entire rid space is exhausted.

Only from were the next index started from.

Quoted 5 lines

I will post a patch.

- Ken

#6 lore raw
In reply to Peter Keilty

Peter Keilty wrote on Friday, October 28, 2005 7:50 AM

Quoted 1 line

The original code did use full range,

Yes or no, first call to wrap_mmu_context occurs at ia64_ctx.next equals 2^15 (32768), then second call occurs at 2097152. First one is needlessly too early. One can say it uses full range. But the real thing I'm after is number of ctx_id allocation in between global tlb flush. Kernel did a global flush before entire 2M ctx_id is used. That is not a "full range" (as in use up all ctx_id before a global tlb flush).

Quoted 4 lines

but once wrapping
occurred yes ranging was used by setting limit. The ranging did go out to the the max_limit on follow on calls, but the range size could small Causing more calls to wrap_mmu_context.

Exactly, ia64_ctx.next can only increment, while ia64_ctx.limit will move down. The code is effectively find_next_hole(), which isn't equivalent to find_largest_hole(). It could have a pathological worst case that you call wrap_mmu_context with only one ctx_id allocation. Worse, since next and limit pair can not cross a wrap around point, when next approach the end of the ctx_id space, the range it find is much smaller at that instance (though that should only occur once every 2M ctx id allocation).

Quoted 6 lines
Quoted 2 lines

Was the lock contention because of much more frequent wrap_mmu_context?

Indirectly, The real reason was the time used to task_list walking derefencing pointers (tens of thousands of processes) trying to find a unused rid.

What is the average number of ctx_id allocation between wrap_mmu_context call? That would tell us how efficient the current find_next_hole() is.

- Ken

#7 lore raw
In reply to Peter Keilty

Peter Keilty wrote on Friday, October 28, 2005 7:50 AM

Quoted 8 lines
Quoted 7 lines

This printk is spurious. When I first read it, my reaction
is: rid as in region register is architecturally defined at maximum 24 bits. Printing 21-bit for rid is misleading. We currently use 8 rid number per process.
So 21 bit for context number is the max. max_ctx will never exceed 1<<21.
Suggest this printk to be dropped.

Quoted 5 lines

I add that per mail with Tony, He suggested that we might limit the bitmap size so that if the rid size was increase in future hardware that a Giga byte Would not be allocated. The printk was to let people know that the rid was being limited to 2^21 if the rid was enlarged.

If I were you, I would try to convince Tony with the argument I presented in my earlier post :-)

- Ken

#8 lore raw
In reply to Peter Keilty

Chen, Kenneth W wrote on Thursday, October 27, 2005 8:09 PM

Quoted 14 lines

Here is the patch, on top of Peter's patch: Add a flush bitmap to track which rid can be recycled when wrap happens. This optimization allows kernel to only wrap and flush tlb when the entire rid space is exhausted. It should dramatically reduce number of rid wrap frequency compare to current implementation.

Lightly tested, I will do more thorough testing. Also, I have a few things want to look at, especially in the area of setting the flushmap bit. There are a few other areas to fine tune Peter's original patch as well.

include/asm-ia64/tlbflush.h flush_tlb_mm (struct mm_struct *mm) · line 56

Quoted 7 lines
include/asm-ia64/tlbflush.h renamed from include/asm-ia64/tlbflush.h.orig +2 -1
@@ -51,7 +51,8 @@ flush_tlb_mm (struct mm_struct *mm)
 	if (!mm)
 		return;
 
-	clear_bit(mm->context, ia64_ctx.bitmap);
+	/* fix me: should we hold ia64_ctx.lock? */
+	set_bit(mm->context, ia64_ctx.flushmap);
 	mm->context = 0;

I convinced myself that I really need a lock here. Otherwise, It will have write-after-write race with flushmap and cause ctx_id to leak. Spin lock is the wrong kind of lock to use here because it should allow concurrent updates between wrap_mmu_context().

arch/ia64/mm/tlb.c renamed from arch/ia64/mm/tlb.c.orig +4 -0
@@ -35,6 +35,7 @@ static struct {
 
 struct ia64_ctx ia64_ctx = {
 	.lock =		SPIN_LOCK_UNLOCKED,
+	.flushmap_rwlock = RW_LOCK_UNLOCKED,
 	.next =		1,
 	.max_ctx =	~0U
 };
@@ -73,9 +74,12 @@ wrap_mmu_context (struct mm_struct *mm)
 {
 	int i;
 
+	write_lock(&ia64_ctx.flushmap_rwlock);
 	bitmap_xor(ia64_ctx.bitmap, ia64_ctx.bitmap,
 		   ia64_ctx.flushmap, ia64_ctx.max_ctx);
 	bitmap_zero(ia64_ctx.flushmap, ia64_ctx.max_ctx);
+	write_unlock(&ia64_ctx.flushmap_rwlock);
+
 	/* use offset at 300 to skip daemons */
 	ia64_ctx.next = find_next_zero_bit(ia64_ctx.bitmap,
 				ia64_ctx.max_ctx, 300);
include/asm-ia64/mmu_context.h renamed from include/asm-ia64/mmu_context.h.orig +2 -1
@@ -31,11 +31,12 @@
 
 struct ia64_ctx {
 	spinlock_t lock;
+	rwlock_t   flushmap_rwlock;
 	unsigned int next;	/* next context number to use */
 	unsigned int max_ctx;	/* max. context value supported by all CPUs */
 				/* next > max_ctx => must call wrap_mmu_context() */
 	unsigned long *bitmap;	/* bitmap size is max_ctx+1 */
-	unsigned long *flushmap;/* pending rid to be flushed */
+	unsigned long *flushmap;/* pending ctx id to be flushed */
 };
 
 extern struct ia64_ctx ia64_ctx;
include/asm-ia64/tlbflush.h renamed from include/asm-ia64/tlbflush.h.orig +3 -2
@@ -52,8 +52,9 @@ flush_tlb_mm (struct mm_struct *mm)
 	if (!mm)
 		return;
 
-	/* fix me: should we hold ia64_ctx.lock? */
-	set_bit(mm->context, ia64_ctx.flushmap);
+	read_lock(&ia64_ctx.flushmap_rwlock);
+	set_bit(mm->context, ia64_ctx.flushmap);
+	read_unlock(&ia64_ctx.flushmap_rwlock);
 	mm->context = 0;
 
 	if (atomic_read(&mm->mm_users) = 0)
#9 lore raw
In reply to Peter Keilty

I did a crude fork bomb test, then printed out ia64_ctx.next ia64_ctx.limit and the diff in wrap_mmu_context(). Here is the result:

First call resulted expected 2M hole. Look what happens after first wrap. My worst fear of global tlb flush after just one ctx_id allocation is actually happening! Average ~ 90,000 ctx_id allocation between flush. Much smaller than advertised 2^21. This find_next_hole() is horribly broken in terms of performance.

- Ken

wrap_mmu_context: next = 32767, limit = 2097152 n = 2064385
wrap_mmu_context: next = 300, limit = 1782 n = 1482
wrap_mmu_context: next = 1783, limit = 1872 n = 89
wrap_mmu_context: next = 1873, limit = 1883 n = 10
wrap_mmu_context: next = 1884, limit = 1921 n = 37
wrap_mmu_context: next = 1922, limit = 1973 n = 51
wrap_mmu_context: next = 1974, limit = 1998 n = 24
wrap_mmu_context: next = 1999, limit = 2055 n = 56
wrap_mmu_context: next = 2056, limit = 2109 n = 53
wrap_mmu_context: next = 2110, limit = 2111 n = 1
wrap_mmu_context: next = 2118, limit = 2124 n = 6
wrap_mmu_context: next = 2125, limit = 2128 n = 3
wrap_mmu_context: next = 2129, limit = 140411 n = 138282
wrap_mmu_context: next = 140412, limit = 518442 n = 378030
wrap_mmu_context: next = 518443, limit = 520789 n = 2346
wrap_mmu_context: next = 520790, limit = 980204 n = 459414
wrap_mmu_context: next = 980205, limit = 1864365 n = 884160
wrap_mmu_context: next = 1864366, limit = 1864528 n = 162
wrap_mmu_context: next = 1864529, limit = 2066917 n = 202388
wrap_mmu_context: next = 2066918, limit = 2097152 n = 30234
wrap_mmu_context: next = 300, limit = 1782 n = 1482
wrap_mmu_context: next = 1783, limit = 1872 n = 89
wrap_mmu_context: next = 1873, limit = 1883 n = 10
wrap_mmu_context: next = 1884, limit = 1921 n = 37
wrap_mmu_context: next = 1922, limit = 1973 n = 51
wrap_mmu_context: next = 1974, limit = 1998 n = 24
wrap_mmu_context: next = 1999, limit = 2055 n = 56
wrap_mmu_context: next = 2056, limit = 2109 n = 53
wrap_mmu_context: next = 2110, limit = 2111 n = 1
wrap_mmu_context: next = 2118, limit = 2124 n = 6
wrap_mmu_context: next = 2125, limit = 2128 n = 3
wrap_mmu_context: next = 2129, limit = 140411 n = 138282
wrap_mmu_context: next = 140412, limit = 184686 n = 44274
wrap_mmu_context: next = 184687, limit = 188664 n = 3977
wrap_mmu_context: next = 188665, limit = 398496 n = 209831
wrap_mmu_context: next = 398497, limit = 492604 n = 94107
wrap_mmu_context: next = 492605, limit = 510747 n = 18142
wrap_mmu_context: next = 510748, limit = 510923 n = 175
wrap_mmu_context: next = 510924, limit = 518442 n = 7518
wrap_mmu_context: next = 518443, limit = 520789 n = 2346
wrap_mmu_context: next = 520790, limit = 980204 n = 459414
wrap_mmu_context: next = 980205, limit = 1864365 n = 884160
wrap_mmu_context: next = 1864366, limit = 1864528 n = 162
wrap_mmu_context: next = 1864529, limit = 2097152 n = 232623
wrap_mmu_context: next = 300, limit = 1782 n = 1482
wrap_mmu_context: next = 1783, limit = 1872 n = 89
wrap_mmu_context: next = 1873, limit = 1883 n = 10
wrap_mmu_context: next = 1884, limit = 1921 n = 37
wrap_mmu_context: next = 1922, limit = 1973 n = 51
wrap_mmu_context: next = 1974, limit = 1998 n = 24
wrap_mmu_context: next = 1999, limit = 2055 n = 56
wrap_mmu_context: next = 2056, limit = 2109 n = 53
wrap_mmu_context: next = 2110, limit = 2111 n = 1
wrap_mmu_context: next = 2118, limit = 2124 n = 6
wrap_mmu_context: next = 2125, limit = 2128 n = 3
wrap_mmu_context: next = 2129, limit = 140411 n = 138282
wrap_mmu_context: next = 140412, limit = 184686 n = 44274
wrap_mmu_context: next = 184687, limit = 188664 n = 3977
wrap_mmu_context: next = 188665, limit = 398496 n = 209831
wrap_mmu_context: next = 398497, limit = 492604 n = 94107
wrap_mmu_context: next = 492605, limit = 510747 n = 18142
wrap_mmu_context: next = 510748, limit = 510923 n = 175
wrap_mmu_context: next = 510924, limit = 518442 n = 7518
wrap_mmu_context: next = 518443, limit = 520789 n = 2346
wrap_mmu_context: next = 520790, limit = 980204 n = 459414
wrap_mmu_context: next = 980205, limit = 1864365 n = 884160
wrap_mmu_context: next = 1864366, limit = 1864528 n = 162
wrap_mmu_context: next = 1864529, limit = 2097152 n = 232623

#10 lore raw
In reply to Peter Keilty

Ken,

It well be worth doing find_next_zero_bit from get_mmu_context Per your patch instead of the global tbl flush.

Quoted 91 lines

-----Original Message-----
From: Chen, Kenneth W [mailto:[email protected]]
Sent: Friday, October 28, 2005 2:41 PM
To: 'Peter Keilty'; [email protected]
Cc: 'Eric Whitney'
Subject: RE: ia64 get_mmu_context patch

I did a crude fork bomb test, then printed out ia64_ctx.next ia64_ctx.limit and the diff in wrap_mmu_context(). Here is the result:

First call resulted expected 2M hole. Look what happens after first wrap. My worst fear of global tlb flush after just one ctx_id allocation is actually happening! Average ~ 90,000 ctx_id allocation between flush. Much smaller than advertised 2^21. This find_next_hole() is horribly broken in terms of performance.

- Ken

wrap_mmu_context: next = 32767, limit = 2097152 n = 2064385
wrap_mmu_context: next = 300, limit = 1782 n = 1482
wrap_mmu_context: next = 1783, limit = 1872 n = 89
wrap_mmu_context: next = 1873, limit = 1883 n = 10
wrap_mmu_context: next = 1884, limit = 1921 n = 37
wrap_mmu_context: next = 1922, limit = 1973 n = 51
wrap_mmu_context: next = 1974, limit = 1998 n = 24
wrap_mmu_context: next = 1999, limit = 2055 n = 56
wrap_mmu_context: next = 2056, limit = 2109 n = 53
wrap_mmu_context: next = 2110, limit = 2111 n = 1
wrap_mmu_context: next = 2118, limit = 2124 n = 6
wrap_mmu_context: next = 2125, limit = 2128 n = 3
wrap_mmu_context: next = 2129, limit = 140411 n = 138282
wrap_mmu_context: next = 140412, limit = 518442 n = 378030
wrap_mmu_context: next = 518443, limit = 520789 n = 2346
wrap_mmu_context: next = 520790, limit = 980204 n = 459414
wrap_mmu_context: next = 980205, limit = 1864365 n = 884160
wrap_mmu_context: next = 1864366, limit = 1864528 n = 162
wrap_mmu_context: next = 1864529, limit = 2066917 n = 202388
wrap_mmu_context: next = 2066918, limit = 2097152 n = 30234
wrap_mmu_context: next = 300, limit = 1782 n = 1482
wrap_mmu_context: next = 1783, limit = 1872 n = 89
wrap_mmu_context: next = 1873, limit = 1883 n = 10
wrap_mmu_context: next = 1884, limit = 1921 n = 37
wrap_mmu_context: next = 1922, limit = 1973 n = 51
wrap_mmu_context: next = 1974, limit = 1998 n = 24
wrap_mmu_context: next = 1999, limit = 2055 n = 56
wrap_mmu_context: next = 2056, limit = 2109 n = 53
wrap_mmu_context: next = 2110, limit = 2111 n = 1
wrap_mmu_context: next = 2118, limit = 2124 n = 6
wrap_mmu_context: next = 2125, limit = 2128 n = 3
wrap_mmu_context: next = 2129, limit = 140411 n = 138282
wrap_mmu_context: next = 140412, limit = 184686 n = 44274
wrap_mmu_context: next = 184687, limit = 188664 n = 3977
wrap_mmu_context: next = 188665, limit = 398496 n = 209831
wrap_mmu_context: next = 398497, limit = 492604 n = 94107
wrap_mmu_context: next = 492605, limit = 510747 n = 18142
wrap_mmu_context: next = 510748, limit = 510923 n = 175
wrap_mmu_context: next = 510924, limit = 518442 n = 7518
wrap_mmu_context: next = 518443, limit = 520789 n = 2346
wrap_mmu_context: next = 520790, limit = 980204 n = 459414
wrap_mmu_context: next = 980205, limit = 1864365 n = 884160
wrap_mmu_context: next = 1864366, limit = 1864528 n = 162
wrap_mmu_context: next = 1864529, limit = 2097152 n = 232623
wrap_mmu_context: next = 300, limit = 1782 n = 1482
wrap_mmu_context: next = 1783, limit = 1872 n = 89
wrap_mmu_context: next = 1873, limit = 1883 n = 10
wrap_mmu_context: next = 1884, limit = 1921 n = 37
wrap_mmu_context: next = 1922, limit = 1973 n = 51
wrap_mmu_context: next = 1974, limit = 1998 n = 24
wrap_mmu_context: next = 1999, limit = 2055 n = 56
wrap_mmu_context: next = 2056, limit = 2109 n = 53
wrap_mmu_context: next = 2110, limit = 2111 n = 1
wrap_mmu_context: next = 2118, limit = 2124 n = 6
wrap_mmu_context: next = 2125, limit = 2128 n = 3
wrap_mmu_context: next = 2129, limit = 140411 n = 138282
wrap_mmu_context: next = 140412, limit = 184686 n = 44274
wrap_mmu_context: next = 184687, limit = 188664 n = 3977
wrap_mmu_context: next = 188665, limit = 398496 n = 209831
wrap_mmu_context: next = 398497, limit = 492604 n = 94107
wrap_mmu_context: next = 492605, limit = 510747 n = 18142
wrap_mmu_context: next = 510748, limit = 510923 n = 175
wrap_mmu_context: next = 510924, limit = 518442 n = 7518
wrap_mmu_context: next = 518443, limit = 520789 n = 2346
wrap_mmu_context: next = 520790, limit = 980204 n = 459414
wrap_mmu_context: next = 980205, limit = 1864365 n = 884160
wrap_mmu_context: next = 1864366, limit = 1864528 n = 162
wrap_mmu_context: next = 1864529, limit = 2097152 n = 232623