Skip to contentSkip to conversation

irq/affinity: Fix extra vecs calculation

PATCH In review Keith Busch 3 participants 9 messages Apr 13, 2017 – Apr 19, 2017 IRQ

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 irq/affinity: Fix extra vecs calculation

This fixes a math error calculating the extra_vecs. The error assumed only 1 cpu per vector, but the value needs to account for the actual number of cpus per vector in order to get the correct remainder for extra CPU assignment.

Fixes 7bf8222b9bd0 ("irq/affinity: Fix CPU spread for unbalanced nodes")
---
 kernel/irq/affinity.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/irq/affinity.c +1 -1
index dc52911..d052947 100644
@@ -108,7 +108,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 		vecs_to_assign = min(vecs_per_node, ncpus);
 
 		/* Account for rounding errors */
-		extra_vecs = ncpus - vecs_to_assign;
+		extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign);
 
 		for (v = 0; curvec < last_affv && v < vecs_to_assign;
 		     curvec++, v++) {
Signature · 1 line
2.7.2
#1 lore raw
irq/affinity: Fix extra vecs calculation
Commit-ID:  3412386b531244f24a27c79ee003506a52a00848
Gitweb:     http://git.kernel.org/tip/3412386b531244f24a27c79ee003506a52a00848
Author:     Keith Busch <[email protected]>
AuthorDate: Thu, 13 Apr 2017 13:28:12 -0400
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Thu, 13 Apr 2017 23:41:00 +0200

irq/affinity: Fix extra vecs calculation

This fixes a math error calculating the extra_vecs. The error assumed only 1 cpu per vector, but the value needs to account for the actual number of cpus per vector in order to get the correct remainder for extra CPU assignment.

Fixes 7bf8222b9bd0 ("irq/affinity: Fix CPU spread for unbalanced nodes")
---
 kernel/irq/affinity.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/irq/affinity.c +1 -1
index dc52911..d052947 100644
@@ -108,7 +108,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 		vecs_to_assign = min(vecs_per_node, ncpus);
 
 		/* Account for rounding errors */
-		extra_vecs = ncpus - vecs_to_assign;
+		extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign);
 
 		for (v = 0; curvec < last_affv && v < vecs_to_assign;
 		     curvec++, v++) {
#2 lore raw
In reply to Keith Busch

Hi,

Something is wrong with this patch. We run CRIU tests for upstream kernels. And we found that a kernel with this patch can't be booted.

https://travis-ci.org/avagin/linux/builds/223557750

We don't have access to console logs and I can't reproduce this issue on my nodes. I tired to revert this patch and everything works as expected.

https://travis-ci.org/avagin/linux/builds/223594172

Here is another report about this patch
https://lkml.org/lkml/2017/4/16/344

Thanks,
Andrei

Quoted 25 lines · Keith Busch

This fixes a math error calculating the extra_vecs. The error assumed only 1 cpu per vector, but the value needs to account for the actual number of cpus per vector in order to get the correct remainder for extra CPU assignment.

Fixes 7bf8222b9bd0 ("irq/affinity: Fix CPU spread for unbalanced nodes")
---
 kernel/irq/affinity.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/irq/affinity.c +1 -1
index dc52911..d052947 100644
@@ -108,7 +108,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 		vecs_to_assign = min(vecs_per_node, ncpus);
 
 		/* Account for rounding errors */
-		extra_vecs = ncpus - vecs_to_assign;
+		extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign);
 
 		for (v = 0; curvec < last_affv && v < vecs_to_assign;
 		     curvec++, v++) {
#3 lore raw
Quoted 7 lines · Andrei Vagin

Hi,

Something is wrong with this patch. We run CRIU tests for upstream kernels. And we found that a kernel with this patch can't be booted.

https://travis-ci.org/avagin/linux/builds/223557750

Quoted 7 lines

We don't have access to console logs and I can't reproduce this issue on my nodes. I tired to revert this patch and everything works as expected.

https://travis-ci.org/avagin/linux/builds/223594172

Here is another report about this patch
https://lkml.org/lkml/2017/4/16/344

Yikes, okay, I've made a mistake somewhere. Sorry about that, I will look into this ASAP.

If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.

---

kernel/irq/affinity.c +3 -0
index d052947..80c45d0 100644
@@ -105,6 +105,9 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 
 		/* Calculate the number of cpus per vector */
 		ncpus = cpumask_weight(nmsk);
+		if (!ncpus)
+			continue;
+
 		vecs_to_assign = min(vecs_per_node, ncpus);
 
 		/* Account for rounding errors */

--

#4 lore raw
Quoted 11 lines · Keith Busch
Quoted 10 lines · Andrei Vagin

Hi,

Something is wrong with this patch. We run CRIU tests for upstream kernels. And we found that a kernel with this patch can't be booted.

https://travis-ci.org/avagin/linux/builds/223557750

We don't have access to console logs and I can't reproduce this issue on my nodes. I tired to revert this patch and everything works as expected.

Quoted 7 lines
Quoted 4 lines

https://travis-ci.org/avagin/linux/builds/223594172

Here is another report about this patch
https://lkml.org/lkml/2017/4/16/344

Yikes, okay, I've made a mistake somewhere. Sorry about that, I will look into this ASAP.

Thank you

Quoted 5 lines

If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.

This patch doesn't fix my problem
https://travis-ci.org/avagin/linux/builds/223674690

Quoted 17 lines

---

kernel/irq/affinity.c +3 -1
index d052947..80c45d0 100644
@@ -105,6 +105,9 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 
 		/* Calculate the number of cpus per vector */
 		ncpus = cpumask_weight(nmsk);
+		if (!ncpus)
+			continue;
+
 		vecs_to_assign = min(vecs_per_node, ncpus);
 
 		/* Account for rounding errors */
--
#5 lore raw
In reply to Keith Busch
Quoted 16 lines · Keith Busch
Quoted 7 lines · Andrei Vagin

Hi,

Something is wrong with this patch. We run CRIU tests for upstream kernels. And we found that a kernel with this patch can't be booted.

https://travis-ci.org/avagin/linux/builds/223557750

Quoted 7 lines

We don't have access to console logs and I can't reproduce this issue on my nodes. I tired to revert this patch and everything works as expected.

https://travis-ci.org/avagin/linux/builds/223594172

Here is another report about this patch
https://lkml.org/lkml/2017/4/16/344

Quoted 7 lines

Yikes, okay, I've made a mistake somewhere. Sorry about that, I will look into this ASAP.

If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.

I printed variables from my test host, I think this can help to investigate the issue:

irq_create_affinity_masks:116: vecs_to_assign 0 ncpus 2 extra_vecs 2 vecs_per_node 0 affv 2 curvec 2 nodes 1

and here is a patch which I use to print them:

kernel/irq/affinity.c +4 -1
index a073a6e..c43c85d 100644
@@ -110,7 +110,10 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
                vecs_to_assign = min(vecs_per_node, ncpus);
 
                /* Account for rounding errors */
-               extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign);
+//             extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign);
+               extra_vecs = ncpus - vecs_to_assign;
+               printk("%s:%d: vecs_to_assign %d ncpus %d extra_vecs %d vecs_per_node %d affv %d curvec %d nodes %d\n",
+                       __func__, __LINE__, vecs_to_assign, ncpus, extra_vecs, vecs_per_node, affv, curvec, nodes);
 
                for (v = 0; curvec < last_affv && v < vecs_to_assign;
                     curvec++, v++) {
Quoted 17 lines

---

kernel/irq/affinity.c +3 -1
index d052947..80c45d0 100644
@@ -105,6 +105,9 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 
 		/* Calculate the number of cpus per vector */
 		ncpus = cpumask_weight(nmsk);
+		if (!ncpus)
+			continue;
+
 		vecs_to_assign = min(vecs_per_node, ncpus);
 
 		/* Account for rounding errors */
--
#6 lore raw
Quoted 10 lines · Andrei Vagin
Quoted 4 lines · Keith Busch

If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.

I printed variables from my test host, I think this can help to investigate the issue:

irq_create_affinity_masks:116: vecs_to_assign 0 ncpus 2 extra_vecs 2 vecs_per_node 0 affv 2 curvec 2 nodes 1

That explains a lot. This setup wants 2 "pre_vectors", but I didn't know that was even a thing. This should fix it:

---

kernel/irq/affinity.c +1 -2
index d052947..eb8b689 100644
@@ -98,13 +98,16 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 		int ncpus, v, vecs_to_assign, vecs_per_node;
 
 		/* Spread the vectors per node */
-		vecs_per_node = (affv - curvec) / nodes;
+		vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;
 
 		/* Get the cpus on this node which are in the mask */
 		cpumask_and(nmsk, cpu_online_mask, cpumask_of_node(n));
--
#7 lore raw
Quoted 7 lines · Keith Busch
Quoted 6 lines · Andrei Vagin
Quoted 4 lines · Keith Busch

If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.

Quoted 7 lines
Quoted 4 lines

I printed variables from my test host, I think this can help to investigate the issue:

irq_create_affinity_masks:116: vecs_to_assign 0 ncpus 2 extra_vecs 2 vecs_per_node 0 affv 2 curvec 2 nodes 1

That explains a lot. This setup wants 2 "pre_vectors", but I didn't know that was even a thing. This should fix it:

This patch works for me.

Quoted 16 lines

---

kernel/irq/affinity.c +1 -2
index d052947..eb8b689 100644
@@ -98,13 +98,16 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 		int ncpus, v, vecs_to_assign, vecs_per_node;
 
 		/* Spread the vectors per node */
-		vecs_per_node = (affv - curvec) / nodes;
+		vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;
 
 		/* Get the cpus on this node which are in the mask */
 		cpumask_and(nmsk, cpu_online_mask, cpumask_of_node(n));
--
#8 lore raw
Quoted 1 line · Andrei Vagin

This patch works for me.

Awesome, thank you much for confirming, and again, sorry for the breakage. I see virtio-scsi is one reliable way to have reproduced this, so I'll incorporate that into tests before posting future kernel core patches.

I'll send this fix with change log shortly.

Quoted 15 lines
Quoted 15 lines

---

kernel/irq/affinity.c +1 -2
index d052947..eb8b689 100644
@@ -98,13 +98,16 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)
 		int ncpus, v, vecs_to_assign, vecs_per_node;
 
 		/* Spread the vectors per node */
-		vecs_per_node = (affv - curvec) / nodes;
+		vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;
 
 		/* Get the cpus on this node which are in the mask */
 		cpumask_and(nmsk, cpu_online_mask, cpumask_of_node(n));
--