Skip to contentSkip to conversation

x86, irq, fix logical AND/OR error in check_irq_vectors_for_cpu_disable()

PATCH In review Prarit Bhargava 4 participants 8 messages Dec 23, 2013 – Dec 28, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86, irq, fix logical AND/OR error in check_irq_vectors_for_cpu_disable()

Patch is against linux-tip.git and was tested on both linux.git and tip without any issues. As expected, the number of required vectors for the down'd cpu drops from 202 to 181 on my test system (which has 509 vectors assigned in total).

Many thanks to Gong Chen for catching this.

P.

----8<----

Gong Chen caught this coding error during inspection of the patch. The code should be AND not OR.

Cc Michel Lespinasse <[email protected]>
Cc Seiji Aguchi <[email protected]>
Cc Yang Zhang <[email protected]>
Cc Paul Gortmaker <[email protected]>
Cc Janet Morgan <[email protected]>
Cc Tony Luck <[email protected]>
Cc Ruiv Wang <[email protected]>
Cc Gong Chen <[email protected]>
Cc Andi Kleen <[email protected]>
Cc H. Peter Anvin <[email protected]>
---
 arch/x86/kernel/irq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/irq.c +1 -1
index 7d40698..aed7acc 100644
@@ -281,7 +281,7 @@ int check_irq_vectors_for_cpu_disable(void)
 			desc = irq_to_desc(irq);
 			data = irq_desc_get_irq_data(desc);
 			affinity = data->affinity;
-			if (irq_has_action(irq) || !irqd_is_per_cpu(data) ||
+			if (irq_has_action(irq) && !irqd_is_per_cpu(data) &&
 			    !cpumask_subset(affinity, cpu_online_mask))
 				this_count++;
 		}
Signature · 1 line
1.8.3.1
#1 lore raw

arch/x86/kernel/irq.c int check_irq_vectors_for_cpu_disable(void) · line 286

Quoted 12 lines · Prarit Bhargava
arch/x86/kernel/irq.c +1 -1
index 7d40698..aed7acc 100644
@@ -281,7 +281,7 @@ int check_irq_vectors_for_cpu_disable(void)
 			desc = irq_to_desc(irq);
 			data = irq_desc_get_irq_data(desc);
 			affinity = data->affinity;
-			if (irq_has_action(irq) || !irqd_is_per_cpu(data) ||
+			if (irq_has_action(irq) && !irqd_is_per_cpu(data) &&
 			    !cpumask_subset(affinity, cpu_online_mask))
 				this_count++;

Hi, Prarit

I noticed that you don't mention another question I asked in last mail.

"It looks like cpu_online_mask will be updated until cpu_disable_common is called, but your check_vectors is called before that."

native_cpu_disable

        cpu_disable_common
                remove_cpu_from_maps
                        /*
                         * until here, cpu_online_mask/cpu_online_bits
                         * is cleared
                         */
                        set_cpu_online(cpu, false);

Something I missed?

#2 lore raw
Quoted 15 lines · Gong

arch/x86/kernel/irq.c int check_irq_vectors_for_cpu_disable(void) · line 286

Quoted 12 lines · Prarit Bhargava
arch/x86/kernel/irq.c +1 -1
index 7d40698..aed7acc 100644
@@ -281,7 +281,7 @@ int check_irq_vectors_for_cpu_disable(void)
 			desc = irq_to_desc(irq);
 			data = irq_desc_get_irq_data(desc);
 			affinity = data->affinity;
-			if (irq_has_action(irq) || !irqd_is_per_cpu(data) ||
+			if (irq_has_action(irq) && !irqd_is_per_cpu(data) &&
 			    !cpumask_subset(affinity, cpu_online_mask))
 				this_count++;

Hi, Prarit

Quoted 4 lines

I noticed that you don't mention another question I asked in last mail.

"It looks like cpu_online_mask will be updated until cpu_disable_common is called, but your check_vectors is called before that."

Oh, I'm sorry ... Yes, check_irq_vectors_for_cpu_disable() is called before we remove the CPU from the maps. If there is an error then we have to do much less clean up of the code. I explicitly take into account the cpu that is being downed into the check vectors code.

Quoted 11 lines

native_cpu_disable

        cpu_disable_common
                remove_cpu_from_maps
                        /*
                         * until here, cpu_online_mask/cpu_online_bits
                         * is cleared
                         */
                        set_cpu_online(cpu, false);

Something I missed?

No -- are you pointing out a bug?

P.

#3 lore raw
Quoted 18 lines · Prarit Bhargava
Quoted 17 lines · Gong

arch/x86/kernel/irq.c int check_irq_vectors_for_cpu_disable(void) · line 286

Quoted 12 lines · Prarit Bhargava
arch/x86/kernel/irq.c +1 -1
index 7d40698..aed7acc 100644
@@ -281,7 +281,7 @@ int check_irq_vectors_for_cpu_disable(void)
 			desc = irq_to_desc(irq);
 			data = irq_desc_get_irq_data(desc);
 			affinity = data->affinity;
-			if (irq_has_action(irq) || !irqd_is_per_cpu(data) ||
+			if (irq_has_action(irq) && !irqd_is_per_cpu(data) &&
 			    !cpumask_subset(affinity, cpu_online_mask))
 				this_count++;

Hi, Prarit

I noticed that you don't mention another question I asked in last mail.

Quoted 8 lines
Quoted 2 lines

"It looks like cpu_online_mask will be updated until cpu_disable_common is called, but your check_vectors is called before that."

Oh, I'm sorry ... Yes, check_irq_vectors_for_cpu_disable() is called before we remove the CPU from the maps. If there is an error then we have to do much less clean up of the code. I explicitly take into account the cpu that is being downed into the check vectors code.

Here is my question: How to decide this_count can be incrased?
1) it is a valid irq(irq_has_action) 2) it is not percpu irq(!irqd_is_per_cpu)
3) it is not shared with left online cpus(!cpumask_subset)

For item 3, I have some concerns.
Your current codes are called before cpu_disable_common, so affinity and cpu_online_mask are both not updated. BTW, it means your calculation for count is not correct because it concludes one to-be-off-lined cpu

+ for_each_online_cpu(cpu) {
+ if (cpu == smp_processor_id())
+ continue;
+ for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS;
+ vector++) {
+ if (per_cpu(vector_irq, cpu)[vector] < 0)
+ count++;
+ }
+ }

Back to my question, assume cpu1 will be off-lined and one irq affinity is set as (1, 2) -- this irq will be bypassed. Looks good. But if one irq affinity is set as only (1), -- this irq is bypassed, too. Not right!

Furthermore, you even can't use cpumask_subset as evaluation condition, whatever affinity/cpu_online_mask is updated or not. Let me paste the comment of cpumask_subset:
/**
* cpumask_subset - (*src1p & ~*src2p) == 0
* @src1p: the first input
* @src2p: the second input
*
* Returns 1 if *@src1p is a subset of *@src2p, else returns 0 */
Here we can see, even if src1p is an empty set, it still can be considered as the subset of src2p. For our this special case, I mean cpu1 will be off-lined and one irq affinity is set as (1). If this irq affinity is updated to (0), it means no cpu is bound to this irq, but the calculation of cpumask_subset will return true and this irq will be bypassed. For this case, cpumask_empty should be more suitable.

#4 lore raw
Quoted 46 lines · Gong
Quoted 18 lines · Prarit Bhargava
Quoted 17 lines · Gong

arch/x86/kernel/irq.c int check_irq_vectors_for_cpu_disable(void) · line 286

Quoted 12 lines · Prarit Bhargava
arch/x86/kernel/irq.c +1 -1
index 7d40698..aed7acc 100644
@@ -281,7 +281,7 @@ int check_irq_vectors_for_cpu_disable(void)
 			desc = irq_to_desc(irq);
 			data = irq_desc_get_irq_data(desc);
 			affinity = data->affinity;
-			if (irq_has_action(irq) || !irqd_is_per_cpu(data) ||
+			if (irq_has_action(irq) && !irqd_is_per_cpu(data) &&
 			    !cpumask_subset(affinity, cpu_online_mask))
 				this_count++;

Hi, Prarit

I noticed that you don't mention another question I asked in last mail.

Quoted 8 lines
Quoted 2 lines

"It looks like cpu_online_mask will be updated until cpu_disable_common is called, but your check_vectors is called before that."

Oh, I'm sorry ... Yes, check_irq_vectors_for_cpu_disable() is called before we remove the CPU from the maps. If there is an error then we have to do much less clean up of the code. I explicitly take into account the cpu that is being downed into the check vectors code.

Here is my question: How to decide this_count can be incrased?
1) it is a valid irq(irq_has_action) 2) it is not percpu irq(!irqd_is_per_cpu)
3) it is not shared with left online cpus(!cpumask_subset)

For item 3, I have some concerns.
Your current codes are called before cpu_disable_common, so affinity and cpu_online_mask are both not updated. BTW, it means your calculation for count is not correct because it concludes one to-be-off-lined cpu

(quoted diff) +9 -0
+       for_each_online_cpu(cpu) {
+               if (cpu == smp_processor_id())
+                       continue;
+               for (vector = FIRST_EXTERNAL_VECTOR; vector < NR_VECTORS;
+                    vector++) {
+                       if (per_cpu(vector_irq, cpu)[vector] < 0)
+                               count++;
+               }
+       }
Quoted 3 lines

Back to my question, assume cpu1 will be off-lined and one irq affinity is set as (1, 2) -- this irq will be bypassed. Looks good. But if one irq affinity is set as only (1), -- this irq is bypassed, too. Not right!

Oh, yes, this is a bug. ... and as you point out ...

Quoted 17 lines

Furthermore, you even can't use cpumask_subset as evaluation condition, whatever affinity/cpu_online_mask is updated or not. Let me paste the comment of cpumask_subset:
/**
* cpumask_subset - (*src1p & ~*src2p) == 0
* @src1p: the first input
* @src2p: the second input
*
* Returns 1 if *@src1p is a subset of *@src2p, else returns 0 */
Here we can see, even if src1p is an empty set, it still can be considered as the subset of src2p. For our this special case, I mean cpu1 will be off-lined and one irq affinity is set as (1). If this irq affinity is updated to (0), it means no cpu is bound to this irq, but the calculation of cpumask_subset will return true and this irq will be bypassed. For this case, cpumask_empty should be more suitable.

I definitely should have used cpumask_empty(). I'll retest ...

P.

#5 lore raw
Quoted 7 lines · Prarit Bhargava
Quoted 4 lines

Back to my question, assume cpu1 will be off-lined and one irq affinity is set as (1, 2) -- this irq will be bypassed. Looks good. But if one irq affinity is set as only (1), -- this irq is bypassed, too. Not right!

Oh, yes, this is a bug. ... and as you point out ...

Does this mean the patch that is currently in my tree should not be pushed to Linus? It sounds like that to me...

-hpa

#6 lore raw
Quoted 11 lines · H. Peter Anvin
Quoted 7 lines · Prarit Bhargava
Quoted 4 lines

Back to my question, assume cpu1 will be off-lined and one irq affinity is set as (1, 2) -- this irq will be bypassed. Looks good. But if one irq affinity is set as only (1), -- this irq is bypassed, too. Not right!

Oh, yes, this is a bug. ... and as you point out ...

Does this mean the patch that is currently in my tree should not be pushed to Linus? It sounds like that to me...

Yes, hpa -- please drop the patch. I will resubmit an updated patch after testing.

P.

Quoted 3 lines

-hpa

#7 lore raw
In reply to Prarit Bhargava
Quoted 5 lines · Prarit Bhargava

Patch is against linux-tip.git and was tested on both linux.git and tip without any issues. As expected, the number of required vectors for the down'd cpu drops from 202 to 181 on my test system (which has 509 vectors assigned in total).

Quoted 9 lines

Many thanks to Gong Chen for catching this.

P.

----8<----

Gong Chen caught this coding error during inspection of the patch. The code should be AND not OR.

Please include a description that is understandable by someone who isn't familiar with the back story to this patch. The patch descriptions become part of the permanent record of the history of the Linux kernel and are critical to it being understandable.

-hpa