irq/affinity: Fix extra vecs calculation
PATCH 1/1 irq/affinity: Fix extra vecs calculation
This fixes a math error calculating the extra_vecs. The error assumed only 1 cpu per vector, but the value needs to account for the actual number of cpus per vector in order to get the correct remainder for extra CPU assignment.
--- kernel/irq/affinity.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/irq/affinity.c +1 -1
@@ -108,7 +108,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd) vecs_to_assign =min (vecs_per_node, ncpus);/* Account for rounding errors */ - extra_vecs = ncpus - vecs_to_assign; + extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign);for (v =0 ; curvec < last_affv && v < vecs_to_assign; curvec++, v++) {
Signature · 1 line
2.7.2
tip-bot for Keith Busch
merge botirq/affinity: Fix extra vecs calculation
Commit-ID: 3412386b531244f24a27c79ee003506a52a00848 Gitweb: http://git.kernel.org/tip/3412386b531244f24a27c79ee003506a52a00848 Author: Keith Busch <[email protected]> AuthorDate: Thu, 13 Apr 2017 13:28:12 -0400 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 13 Apr 2017 23:41:00 +0200
irq/affinity: Fix extra vecs calculation
This fixes a math error calculating the extra_vecs. The error assumed only 1 cpu per vector, but the value needs to account for the actual number of cpus per vector in order to get the correct remainder for extra CPU assignment.
--- kernel/irq/affinity.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/irq/affinity.c +1 -1
@@ -108,7 +108,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd) vecs_to_assign =min (vecs_per_node, ncpus);/* Account for rounding errors */ - extra_vecs = ncpus - vecs_to_assign; + extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign);for (v =0 ; curvec < last_affv && v < vecs_to_assign; curvec++, v++) {
Hi,
Something is wrong with this patch. We run CRIU tests for upstream kernels. And we found that a kernel with this patch can't be booted.
https://travis-ci.org/avagin/linux/builds/223557750
We don't have access to console logs and I can't reproduce this issue on my nodes. I tired to revert this patch and everything works as expected.
https://travis-ci.org/avagin/linux/builds/223594172
Here is another report about this patch
https://lkml.org/lkml/2017/4/16/344
Thanks,
Andrei
Quoted 25 lines · Keith Busch
This fixes a math error calculating the
extra_vecs. The error assumed only 1 cpu per vector, but the value needs to account for the actual number of cpus per vector in order to get the correct remainder for extra CPU assignment.Fixes 7bf8222b9bd0 ("irq/affinity: Fix CPU spread for unbalanced nodes")Reported-by Xiaolong Ye <[email protected]>Signed-off-by Keith Busch <[email protected]>--- kernel/irq/affinity.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/irq/affinity.c +1
-1@@ -108,7 +108,7 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd) vecs_to_assign =min (vecs_per_node, ncpus);/* Account for rounding errors */ - extra_vecs = ncpus - vecs_to_assign; + extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign);for (v =0 ; curvec < last_affv && v < vecs_to_assign; curvec++, v++) {
Quoted 7 lines · Andrei Vagin
Hi,
Something is wrong with this patch. We run CRIU tests for upstream kernels. And we found that a kernel with this patch can't be booted.
Quoted 7 lines
We don't have access to console logs and I can't reproduce this issue on my nodes. I tired to revert this patch and everything works as expected.
https://travis-ci.org/avagin/linux/builds/223594172
Here is another report about this patch
https://lkml.org/lkml/2017/4/16/344
Yikes, okay, I've made a mistake somewhere. Sorry about that, I will look into this ASAP.
If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.
---
kernel/irq/affinity.c +3 -0
@@ -105,6 +105,9 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)/* Calculate the number of cpus per vector */ ncpus =cpumask_weight (nmsk); +if (!ncpus) +continue ; + vecs_to_assign =min (vecs_per_node, ncpus);/* Account for rounding errors */
--
Quoted 11 lines · Keith Busch
Quoted 10 lines · Andrei Vagin
Hi,
Something is wrong with this patch. We run CRIU tests for upstream kernels. And we found that a kernel with this patch can't be booted.
https://travis-ci.org/avagin/linux/builds/223557750
We don't have access to console logs and I can't reproduce this issue on my nodes. I tired to revert this patch and everything works as expected.
Quoted 7 lines
Quoted 4 lines
https://travis-ci.org/avagin/linux/builds/223594172
Here is another report about this patch
https://lkml.org/lkml/2017/4/16/344Yikes, okay, I've made a mistake somewhere. Sorry about that, I will look into this ASAP.
Thank you
Quoted 5 lines
If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.
This patch doesn't fix my problem
https://travis-ci.org/avagin/linux/builds/223674690
Quoted 17 lines
---
kernel/irq/affinity.c +3
-1@@ -105,6 +105,9 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)/* Calculate the number of cpus per vector */ ncpus =cpumask_weight (nmsk); +if (!ncpus) +continue ; + vecs_to_assign =min (vecs_per_node, ncpus);/* Account for rounding errors */ --
Quoted 16 lines · Keith Busch
Quoted 7 lines · Andrei Vagin
Hi,
Something is wrong with this patch. We run CRIU tests for upstream kernels. And we found that a kernel with this patch can't be booted.
Quoted 7 lines
We don't have access to console logs and I can't reproduce this issue on my nodes. I tired to revert this patch and everything works as expected.
https://travis-ci.org/avagin/linux/builds/223594172
Here is another report about this patch
https://lkml.org/lkml/2017/4/16/344
Quoted 7 lines
Yikes, okay, I've made a mistake somewhere. Sorry about that, I will look into this ASAP.
If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.
I printed variables from my test host, I think this can help to investigate the issue:
irq_create_affinity_masks:116: vecs_to_assign 0 ncpus 2 extra_vecs 2 vecs_per_node 0 affv 2 curvec 2 nodes 1
and here is a patch which I use to print them:
kernel/irq/affinity.c +4 -1
@@ -110,7 +110,10 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd) vecs_to_assign =min (vecs_per_node, ncpus);/* Account for rounding errors */ - extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign); +// extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign); + extra_vecs = ncpus - vecs_to_assign; +printk ("%s:%d: vecs_to_assign %d ncpus %d extra_vecs %d vecs_per_node %d affv %d curvec %d nodes %d\n" , + __func__, __LINE__, vecs_to_assign, ncpus, extra_vecs, vecs_per_node, affv, curvec, nodes);for (v =0 ; curvec < last_affv && v < vecs_to_assign; curvec++, v++) {
Quoted 17 lines
---
kernel/irq/affinity.c +3
-1@@ -105,6 +105,9 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)/* Calculate the number of cpus per vector */ ncpus =cpumask_weight (nmsk); +if (!ncpus) +continue ; + vecs_to_assign =min (vecs_per_node, ncpus);/* Account for rounding errors */ --
Quoted 10 lines · Andrei Vagin
Quoted 4 lines · Keith Busch
If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.
I printed variables from my test host, I think this can help to investigate the issue:
irq_create_affinity_masks:116:
vecs_to_assign0 ncpus 2extra_vecs2vecs_per_node0 affv 2 curvec 2 nodes 1
That explains a lot. This setup wants 2 "pre_vectors", but I didn't know that was even a thing. This should fix it:
---
kernel/irq/affinity.c +1 -2
@@ -98,13 +98,16 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)int ncpus, v, vecs_to_assign, vecs_per_node;/* Spread the vectors per node */ - vecs_per_node = (affv - curvec) / nodes; + vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;/* Get the cpus on this node which are in the mask */ cpumask_and (nmsk, cpu_online_mask,cpumask_of_node (n)); --
Quoted 7 lines · Keith Busch
Quoted 6 lines · Andrei Vagin
Quoted 4 lines · Keith Busch
If it's a divide by 0 as your last link indicates, that must mean there are possible nodes, but have no CPUs, and those should be skipped. If that's the case, the following should fix it, but I'm going to do some more qemu testing with various CPU topologies to confirm.
Quoted 7 lines
Quoted 4 lines
I printed variables from my test host, I think this can help to investigate the issue:
irq_create_affinity_masks:116:
vecs_to_assign0 ncpus 2extra_vecs2vecs_per_node0 affv 2 curvec 2 nodes 1That explains a lot. This setup wants 2 "
pre_vectors", but I didn't know that was even a thing. This should fix it:
This patch works for me.
Quoted 16 lines
---
kernel/irq/affinity.c +1
-2@@ -98,13 +98,16 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)int ncpus, v, vecs_to_assign, vecs_per_node;/* Spread the vectors per node */ - vecs_per_node = (affv - curvec) / nodes; + vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;/* Get the cpus on this node which are in the mask */ cpumask_and (nmsk, cpu_online_mask,cpumask_of_node (n)); --
Quoted 1 line · Andrei Vagin
This patch works for me.
Awesome, thank you much for confirming, and again, sorry for the breakage. I see virtio-scsi is one reliable way to have reproduced this, so I'll incorporate that into tests before posting future kernel core patches.
I'll send this fix with change log shortly.
Quoted 15 lines
Quoted 15 lines
---
kernel/irq/affinity.c +1
-2@@ -98,13 +98,16 @@ irq_create_affinity_masks(int nvecs, const struct irq_affinity *affd)int ncpus, v, vecs_to_assign, vecs_per_node;/* Spread the vectors per node */ - vecs_per_node = (affv - curvec) / nodes; + vecs_per_node = (affv - (curvec - affd->pre_vectors)) / nodes;/* Get the cpus on this node which are in the mask */ cpumask_and (nmsk, cpu_online_mask,cpumask_of_node (n)); --