Skip to contentSkip to conversation

x86_64 irq: reuse vector for __assign_irq_vector

PATCH In review yhlu 4 participants 8 messages Oct 24, 2006 – Oct 28, 2006

lore
Share & tools
lore

yhlu

author
#0 lore raw

PATCH 1/1 x86_64 irq: reuse vector for __assign_irq_vector

in phys flat mode, when using set_xxx_irq_affinity to irq balance from one cpu to another, _assign_irq_vector will get to increase last used vector and get new vector. this will use up the vector if enough set_xxx_irq_affintiy are called. and end with using same vector in different cpu for different irq. (that is not what we want, we only want to use same vector in different cpu for different irq when more than 0x240 irq needed). To keep it simple, the vector should be reused instead of getting new vector.

Also according to Eric's review, make it more generic to be used with flat mode too.

This patch need to be applied over Eric's irq: cpu_online_map patch.

Cc Eric W. Biederman <[email protected]>
Cc Muli Ben-Yehuda <[email protected]>
arch/x86_64/kernel/io_apic.c +43 -21
@@ -613,8 +613,9 @@
 	 * 0x80, because int 0x80 is hm, kind of importantish. ;)
 	 */
 	static int current_vector = FIRST_DEVICE_VECTOR, current_offset = 0;
-	int old_vector = -1;
-	int cpu;
+	int vector = -1, old_vector = -1;
+	cpumask_t domain, new_mask;
+	int cpu, new_cpu;
 
 	BUG_ON((unsigned)irq >= NR_IRQ_VECTORS);
 
@@ -624,15 +625,30 @@
 	if (irq_vector[irq] > 0)
 		old_vector = irq_vector[irq];
 	if (old_vector > 0) {
-		cpus_and(*result, irq_domain[irq], mask);
-		if (!cpus_empty(*result))
-			return old_vector;
+		/* try to reuse vector */
+		for_each_cpu_mask(cpu, mask) {
+			int can_reuse = 1;
+			domain = vector_allocation_domain(cpu);
+			cpus_and(new_mask, domain, cpu_online_map);
+			for_each_cpu_mask(new_cpu, new_mask) {
+				int old_irq;
+				old_irq = per_cpu(vector_irq, new_cpu)[old_vector];
+				if ( (old_irq != irq) && (old_irq != -1)) {
+					can_reuse = 0;
+					break;
+				}
+			}
+
+			if(!can_reuse) continue;
+
+			vector = old_vector;
+			goto found_one;	
+		}
+
 	}
 
 	for_each_cpu_mask(cpu, mask) {
-		cpumask_t domain, new_mask;
-		int new_cpu;
-		int vector, offset;
+		int offset;
 
 		domain = vector_allocation_domain(cpu);
 		cpus_and(new_mask, domain, cpu_online_map);
@@ -656,21 +672,27 @@
 		/* Found one! */
 		current_vector = vector;
 		current_offset = offset;
-		if (old_vector >= 0) {
-			cpumask_t old_mask;
-			int old_cpu;
-			cpus_and(old_mask, irq_domain[irq], cpu_online_map);
-			for_each_cpu_mask(old_cpu, old_mask)
-				per_cpu(vector_irq, old_cpu)[old_vector] = -1;
-		}
-		for_each_cpu_mask(new_cpu, new_mask)
-			per_cpu(vector_irq, new_cpu)[vector] = irq;
-		irq_vector[irq] = vector;
-		irq_domain[irq] = domain;
-		cpus_and(*result, domain, mask);
-		return vector;
+		
+		goto found_one;
 	}
+
 	return -ENOSPC;
+
+found_one:
+	if (old_vector >= 0) {
+		cpumask_t old_mask;
+		int old_cpu;
+		cpus_and(old_mask, irq_domain[irq], cpu_online_map);
+		for_each_cpu_mask(old_cpu, old_mask)
+			per_cpu(vector_irq, old_cpu)[old_vector] = -1;
+	}
+	for_each_cpu_mask(new_cpu, new_mask)
+		per_cpu(vector_irq, new_cpu)[vector] = irq;
+	irq_vector[irq] = vector;
+	irq_domain[irq] = domain;
+	cpus_and(*result, domain, mask);
+	return vector;
+
 }
 
 static int assign_irq_vector(int irq, cpumask_t mask, cpumask_t *result)
#1 lore raw
Quoted 12 lines · yhlu

in phys flat mode, when using set_xxx_irq_affinity to irq balance from one cpu to another, _assign_irq_vector will get to increase last used vector and get new vector. this will use up the vector if enough set_xxx_irq_affintiy are called. and end with using same vector in different cpu for different irq. (that is not what we want, we only want to use same vector in different cpu for different irq when more than 0x240 irq needed). To keep it simple, the vector should be reused instead of getting new vector.

Also according to Eric's review, make it more generic to be used with flat mode too.

Quoted 6 lines

This patch need to be applied over Eric's irq: cpu_online_map patch.

Cc Eric W. Biederman <[email protected]>
Cc Muli Ben-Yehuda <[email protected]>

Boots fine and survives a quick stress test.

Cheers,
Muli

#2 lore raw
In reply to yhlu
Quoted 11 lines · yhlu

in phys flat mode, when using set_xxx_irq_affinity to irq balance from one cpu to another, _assign_irq_vector will get to increase last used vector and get new vector. this will use up the vector if enough set_xxx_irq_affintiy are called. and end with using same vector in different cpu for different irq. (that is not what we want, we only want to use same vector in different cpu for different irq when more than 0x240 irq needed). To keep it simple, the vector should be reused instead of getting new vector.

Also according to Eric's review, make it more generic to be used with flat mode too.

Is that still needed with Eric's latest patches? I suppose not?

-Andi

#3 lore raw

Andi,
please check the revised patch: It can be appied to current linus't tree.

in phys flat mode, when using set_xxx_irq_affinity to irq balance from one cpu to another, _assign_irq_vector will get to increase last used vector and get new vector. this will use up the vector if enough set_xxx_irq_affintiy are called. and end with using same vector in different cpu for different irq. (that is not what we want, we only want to use same vector in different cpu for different irq when more than 0x240 irq needed). To keep it simple, the vector should be reused instead of getting new vector.

Also according to Eric's review, make it more generic to be used with flat mode too.

It also check if new domain and old domain is equal, to avoid extra operation.

Cc Eric W. Biederman <[email protected]>
Cc Muli Ben-Yehuda <[email protected]>
arch/x86_64/kernel/io_apic.c +51 -21
index fe429e5..51535e8 100644
@@ -613,8 +613,9 @@ static int __assign_irq_vector(int irq, 
 	 * 0x80, because int 0x80 is hm, kind of importantish. ;)
 	 */
 	static int current_vector = FIRST_DEVICE_VECTOR, current_offset = 0;
-	int old_vector = -1;
-	int cpu;
+	int vector = -1, old_vector = -1;
+	cpumask_t domain, new_mask;
+	int cpu, new_cpu;
 
 	BUG_ON((unsigned)irq >= NR_IRQ_VECTORS);
 
@@ -624,15 +625,38 @@ static int __assign_irq_vector(int irq, 
 	if (irq_vector[irq] > 0)
 		old_vector = irq_vector[irq];
 	if (old_vector > 0) {
-		cpus_and(*result, irq_domain[irq], mask);
-		if (!cpus_empty(*result))
-			return old_vector;
+		/* try to reuse vector */
+		for_each_cpu_mask(cpu, mask) {
+			int can_reuse = 1;
+
+			domain = vector_allocation_domain(cpu);
+			cpus_and(new_mask, domain, cpu_online_map);
+
+			if (cpus_equal(domain, irq_domain[irq])) {
+				cpus_and(*result, irq_domain[irq], mask);
+				if (!cpus_empty(*result))  
+					return old_vector;
+			}
+			
+			for_each_cpu_mask(new_cpu, new_mask) {
+				int old_irq;
+				old_irq = per_cpu(vector_irq, new_cpu)[old_vector];
+				if ( (old_irq != irq) && (old_irq != -1)) {
+					can_reuse = 0;
+					break;
+				}
+			}
+
+			if(!can_reuse) 
+				continue;
+
+			vector = old_vector;
+			goto found_one;	
+		}
 	}
 
 	for_each_cpu_mask(cpu, mask) {
-		cpumask_t domain, new_mask;
-		int new_cpu;
-		int vector, offset;
+		int offset;
 
 		domain = vector_allocation_domain(cpu);
 		cpus_and(new_mask, domain, cpu_online_map);
@@ -656,21 +680,27 @@ next:
 		/* Found one! */
 		current_vector = vector;
 		current_offset = offset;
-		if (old_vector >= 0) {
-			cpumask_t old_mask;
-			int old_cpu;
-			cpus_and(old_mask, irq_domain[irq], cpu_online_map);
-			for_each_cpu_mask(old_cpu, old_mask)
-				per_cpu(vector_irq, old_cpu)[old_vector] = -1;
-		}
-		for_each_cpu_mask(new_cpu, new_mask)
-			per_cpu(vector_irq, new_cpu)[vector] = irq;
-		irq_vector[irq] = vector;
-		irq_domain[irq] = domain;
-		cpus_and(*result, domain, mask);
-		return vector;
+		
+		goto found_one;
 	}
+
 	return -ENOSPC;
+
+found_one:
+	if (old_vector >= 0) {
+		cpumask_t old_mask;
+		int old_cpu;
+		cpus_and(old_mask, irq_domain[irq], cpu_online_map);
+		for_each_cpu_mask(old_cpu, old_mask)
+			per_cpu(vector_irq, old_cpu)[old_vector] = -1;
+	}
+	for_each_cpu_mask(new_cpu, new_mask)
+		per_cpu(vector_irq, new_cpu)[vector] = irq;
+	irq_vector[irq] = vector;
+	irq_domain[irq] = domain;
+	cpus_and(*result, domain, mask);
+	return vector;
+
 }
 
 static int assign_irq_vector(int irq, cpumask_t mask, cpumask_t *result)
#4 lore raw
Quoted 12 lines · Yinghai Lu

Andi,
please check the revised patch: It can be appied to current linus't tree.

in phys flat mode, when using set_xxx_irq_affinity to irq balance from one cpu to another, _assign_irq_vector will get to increase last used vector and get new vector. this will use up the vector if enough set_xxx_irq_affintiy are called. and end with using same vector in different cpu for different irq. (that is not what we want, we only want to use same vector in different cpu for different irq when more than 0x240 irq needed). To keep it simple, the vector should be reused instead of getting new vector.

Quoted 5 lines

Also according to Eric's review, make it more generic to be used with flat mode too.

It also check if new domain and old domain is equal, to avoid extra operation.

Works fine for mee.

Cheers,
Muli

#5 lore raw
In reply to Yinghai Lu
Quoted 12 lines · Yinghai Lu

Andi,
please check the revised patch: It can be appied to current linus't tree.

in phys flat mode, when using set_xxx_irq_affinity to irq balance from one cpu to another, _assign_irq_vector will get to increase last used vector and get new vector. this will use up the vector if enough set_xxx_irq_affintiy are called. and end with using same vector in different cpu for different irq. (that is not what we want, we only want to use same vector in different cpu for different irq when more than 0x240 irq needed). To keep it simple, the vector should be reused instead of getting new vector.

Quoted 2 lines

Also according to Eric's review, make it more generic to be used with flat mode too.

Eric, do you ack the patch?

-andi

#6 lore raw
In reply to yhlu
Quoted 9 lines · yhlu

in phys flat mode, when using set_xxx_irq_affinity to irq balance from one cpu to another, _assign_irq_vector will get to increase last used vector and get new vector. this will use up the vector if enough set_xxx_irq_affintiy are called. and end with using same vector in different cpu for different irq. (that is not what we want, we only want to use same vector in different cpu for different irq when more than 0x240 irq needed). To keep it simple, the vector should be reused instead of getting new vector.

Quoted 3 lines

Also according to Eric's review, make it more generic to be used with flat mode too.

Added thanks

Quoted 1 line

This patch need to be applied over Eric's irq: cpu_online_map patch.

Hmm, i'm not sure I got that. Which was patch was it exactly

Or can you please double check it is ok in 2.6.19rc3 + ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/x86_64-2.6.19-rc3-061028-1.bz2

Thanks,

-Andi

#7 lore raw
Quoted 3 lines · Andi Kleen

On Mon, Oct 23, 2006 at 09:15:31PM -0700, yhlu wrote:

Hmm, i'm not sure I got that. Which was patch was it exactly

Eric's online cpus patch is already in the maintream.

Please use the diff in

http://lkml.org/lkml/2006/10/26/38

Thanks

YH