Skip to contentSkip to conversation

genriq/affinity: two improvement on __irq_build_affinity_masks

PATCH Superseded Ming Lei 3 participants 6 messages Aug 9, 2019 – Aug 12, 2019

Subsystems (2)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

cover genriq/affinity: two improvement on __irq_build_affinity_masks

Hi,

The 1st patch makes __irq_build_affinity_masks() more reliable, such as, all nodes can be covered in the spread.

The 2nd patch spread vectors on node according to the ratio of this node's CPU number to number of all remaining CPUs, then vectors assignment can become more fair. Meantime, the warning report from Jon Derrick can be fixed.

Please review & comment!

Ming Lei (2):
genirq/affinity: improve __irq_build_affinity_masks() genirq/affinity: spread vectors on node according to nr_cpu ratio

 kernel/irq/affinity.c | 46 +++++++++++++++++++++++++++++++------------
 1 file changed, 33 insertions(+), 13 deletions(-)
Cc Christoph Hellwig <hch at lst.de>
Cc Keith Busch <kbusch at kernel.org>
Cc linux-nvme at lists.infradead.org,
Cc Jon Derrick <jonathan.derrick at intel.com>
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH 1/2 genirq/affinity: improve __irq_build_affinity_masks()

One invariant of __irq_build_affinity_masks() is that all CPUs in the specified masks( cpu_mask AND node_to_cpumask for each node) should be covered during the spread. Even though all requested vectors have been reached, we still need to spread vectors among left CPUs. The similar policy has been taken in case of 'numvecs <= nodes'.

So remove the following check inside the loop:

	if (done >= numvecs)
		break;

Meantime assign at least 1 vector for left nodes if 'numvecs' vectors have been spread.

Also, if the specified cpumask for one numa node is empty, simply not spread vectors on this node.

Cc Christoph Hellwig <hch at lst.de>
Cc Keith Busch <kbusch at kernel.org>
Cc linux-nvme at lists.infradead.org,
Cc Jon Derrick <jonathan.derrick at intel.com>
Signed-off-by Ming Lei <ming.lei at redhat.com>
---
 kernel/irq/affinity.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
kernel/irq/affinity.c +21 -12
index 6fef48033f96..bc3652a2c61b 100644
@@ -129,21 +129,32 @@ static int __irq_build_affinity_masks(unsigned int startvec,
 	for_each_node_mask(n, nodemsk) {
 		unsigned int ncpus, v, vecs_to_assign, vecs_per_node;
 
-		/* Spread the vectors per node */
-		vecs_per_node = (numvecs - (curvec - firstvec)) / nodes;
-
 		/* Get the cpus on this node which are in the mask */
 		cpumask_and(nmsk, cpu_mask, node_to_cpumask[n]);
-
-		/* Calculate the number of cpus per vector */
 		ncpus = cpumask_weight(nmsk);
+		if (!ncpus)
+			continue;
+
+		/*
+		 * Calculate the number of cpus per vector
+		 *
+		 * Spread the vectors evenly per node. If the requested
+		 * vector number has been reached, simply allocate one
+		 * vector for each remaining node so that all nodes can
+		 * be covered
+		 */
+		if (numvecs > done)
+			vecs_per_node = max_t(unsigned,
+					(numvecs - done) / nodes, 1);
+		else
+			vecs_per_node = 1;
+
 		vecs_to_assign = min(vecs_per_node, ncpus);
 
 		/* Account for rounding errors */
 		extra_vecs = ncpus - vecs_to_assign * (ncpus / vecs_to_assign);
 
-		for (v = 0; curvec < last_affv && v < vecs_to_assign;
-		     curvec++, v++) {
+		for (v = 0; v < vecs_to_assign; v++) {
 			cpus_per_vec = ncpus / vecs_to_assign;
 
 			/* Account for extra vectors to compensate rounding errors */
@@ -153,16 +164,14 @@ static int __irq_build_affinity_masks(unsigned int startvec,
 			}
 			irq_spread_init_one(&masks[curvec].mask, nmsk,
 						cpus_per_vec);
+			if (++curvec >= last_affv)
+				curvec = firstvec;
 		}
 
 		done += v;
-		if (done >= numvecs)
-			break;
-		if (curvec >= last_affv)
-			curvec = firstvec;
 		--nodes;
 	}
-	return done;
+	return done < numvecs ? done : numvecs;
 }
 
 /*
Signature · 1 line
2.20.1
maintainer
#2 lore raw

Re: [PATCH 1/2] genirq/affinity: improve __irq_build_affinity_masks()

Quoted 34 lines · Ming Lei

One invariant of __irq_build_affinity_masks() is that all CPUs in the specified masks( cpu_mask AND node_to_cpumask for each node) should be covered during the spread. Even though all requested vectors have been reached, we still need to spread vectors among left CPUs. The similar policy has been taken in case of 'numvecs <= nodes'.

So remove the following check inside the loop:

	if (done >= numvecs)
		break;

Meantime assign at least 1 vector for left nodes if 'numvecs' vectors have been spread.

Also, if the specified cpumask for one numa node is empty, simply not spread vectors on this node.

Cc Christoph Hellwig <hch at lst.de>
Cc Keith Busch <kbusch at kernel.org>
Cc linux-nvme at lists.infradead.org,
Cc Jon Derrick <jonathan.derrick at intel.com>
Signed-off-by Ming Lei <ming.lei at redhat.com>
---
 kernel/irq/affinity.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
kernel/irq/affinity.c +0 -0
index 6fef48033f96..bc3652a2c61b 100644
@@ -129,21 +129,32 @@ static int __irq_build_affinity_masks(unsigned int startvec,
 	for_each_node_mask(n, nodemsk) {
 		unsigned int ncpus, v, vecs_to_assign, vecs_per_node;
 

kernel/irq/affinity.c static int __irq_build_affinity_masks(unsigned int startvec, · line 136

Quoted 10 lines
kernel/irq/affinity.c +2 -5
-		/* Spread the vectors per node */
-		vecs_per_node = (numvecs - (curvec - firstvec)) / nodes;
-
 		/* Get the cpus on this node which are in the mask */
 		cpumask_and(nmsk, cpu_mask, node_to_cpumask[n]);
-
-		/* Calculate the number of cpus per vector */
 		ncpus = cpumask_weight(nmsk);
+		if (!ncpus)
+			continue;

This shouldn't be possible, right? The nodemsk we're looping wouldn't have had that node set if no CPUs intersect the node_to_cpu_mask for that node, so the resulting cpumask should always have a non-zero weight.

kernel/irq/affinity.c static int __irq_build_affinity_masks(unsigned int startvec, · line 168

Quoted 6 lines
kernel/irq/affinity.c +2 -0
@@ -153,16 +164,14 @@ static int __irq_build_affinity_masks(unsigned int startvec,
 			}
 			irq_spread_init_one(&masks[curvec].mask, nmsk,
 						cpus_per_vec);
+			if (++curvec >= last_affv)
+				curvec = firstvec;

I'm not so sure about wrapping the vector to share it across nodes. We have enough vectors in this path to ensure each compute node can have a unique one, and it's much cheaper to share these within nodes than across them.

Quoted 12 lines
kernel/irq/affinity.c +1 -5
 		}
 
 		done += v;
-		if (done >= numvecs)
-			break;
-		if (curvec >= last_affv)
-			curvec = firstvec;
 		--nodes;
 	}
-	return done;
+	return done < numvecs ? done : numvecs;
 }
#3 lore raw

Re: [PATCH 1/2] genirq/affinity: improve __irq_build_affinity_masks()

Quoted 36 lines · Keith Busch
Quoted 34 lines · Ming Lei

One invariant of __irq_build_affinity_masks() is that all CPUs in the specified masks( cpu_mask AND node_to_cpumask for each node) should be covered during the spread. Even though all requested vectors have been reached, we still need to spread vectors among left CPUs. The similar policy has been taken in case of 'numvecs <= nodes'.

So remove the following check inside the loop:

if (done >= numvecs)
break;

Meantime assign at least 1 vector for left nodes if 'numvecs' vectors have been spread.

Also, if the specified cpumask for one numa node is empty, simply not spread vectors on this node.

Cc Christoph Hellwig <hch at lst.de>
Cc Keith Busch <kbusch at kernel.org>
Cc linux-nvme at lists.infradead.org,
Cc Jon Derrick <jonathan.derrick at intel.com>
Signed-off-by Ming Lei <ming.lei at redhat.com>
---
 kernel/irq/affinity.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
kernel/irq/affinity.c +0 -0
index 6fef48033f96..bc3652a2c61b 100644
@@ -129,21 +129,32 @@ static int __irq_build_affinity_masks(unsigned int startvec,
      for_each_node_mask(n, nodemsk) {
              unsigned int ncpus, v, vecs_to_assign, vecs_per_node;
 
Quoted 14 lines

kernel/irq/affinity.c static int __irq_build_affinity_masks(unsigned int startvec, · line 136

Quoted 10 lines
kernel/irq/affinity.c +2 -5
-             /* Spread the vectors per node */
-             vecs_per_node = (numvecs - (curvec - firstvec)) / nodes;
-
              /* Get the cpus on this node which are in the mask */
              cpumask_and(nmsk, cpu_mask, node_to_cpumask[n]);
-
-             /* Calculate the number of cpus per vector */
              ncpus = cpumask_weight(nmsk);
+             if (!ncpus)
+                     continue;

This shouldn't be possible, right? The nodemsk we're looping wouldn't have had that node set if no CPUs intersect the node_to_cpu_mask for that node, so the resulting cpumask should always have a non-zero weight.

     cpumask_and(nmsk, cpu_mask, node_to_cpumask[n]);

It is often true, see the following cases:

1) all CPUs in one node are not present

OR

2) all CPUs in one node are present

Quoted 9 lines

kernel/irq/affinity.c static int __irq_build_affinity_masks(unsigned int startvec, · line 168

Quoted 7 lines
kernel/irq/affinity.c +2 -0
@@ -153,16 +164,14 @@ static int __irq_build_affinity_masks(unsigned int startvec,
                      }
                      irq_spread_init_one(&masks[curvec].mask, nmsk,
                                              cpus_per_vec);
+                     if (++curvec >= last_affv)
+                             curvec = firstvec;

I'm not so sure about wrapping the vector to share it across nodes. We

The wrapping is always there, not added by this patch.

Most time it won't happen since we spread vectors on remaining (un-assigned)nodes. And it only happens when there is remaining nodes not spread. We have to make sure all nodes are spread.

And the similar policy is applied on the branch of 'numvecs <= nodes' too.

Quoted 3 lines

have enough vectors in this path to ensure each compute node can have a unique one, and it's much cheaper to share these within nodes than across them.

The patch just moves the wrapping from loop outside into the loop, then all 'extra_vecs' can be covered because it is always < 'vecs_to_assign'.

What matters is that the following check is removed:

if (done >= numvecs)

                break;

then all nodes can be covered.

Thanks,
Ming Lei

#4 lore raw

Re: [PATCH 1/2] genirq/affinity: improve __irq_build_affinity_masks()

Quoted 73 lines · Ming Lei
Quoted 36 lines · Keith Busch
Quoted 34 lines · Ming Lei

One invariant of __irq_build_affinity_masks() is that all CPUs in the specified masks( cpu_mask AND node_to_cpumask for each node) should be covered during the spread. Even though all requested vectors have been reached, we still need to spread vectors among left CPUs. The similar policy has been taken in case of 'numvecs <= nodes'.

So remove the following check inside the loop:

if (done >= numvecs)
break;

Meantime assign at least 1 vector for left nodes if 'numvecs' vectors have been spread.

Also, if the specified cpumask for one numa node is empty, simply not spread vectors on this node.

Cc Christoph Hellwig <hch at lst.de>
Cc Keith Busch <kbusch at kernel.org>
Cc linux-nvme at lists.infradead.org,
Cc Jon Derrick <jonathan.derrick at intel.com>
Signed-off-by Ming Lei <ming.lei at redhat.com>
---
 kernel/irq/affinity.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
kernel/irq/affinity.c +0 -0
index 6fef48033f96..bc3652a2c61b 100644
@@ -129,21 +129,32 @@ static int __irq_build_affinity_masks(unsigned int startvec,
      for_each_node_mask(n, nodemsk) {
              unsigned int ncpus, v, vecs_to_assign, vecs_per_node;
 
Quoted 14 lines

kernel/irq/affinity.c static int __irq_build_affinity_masks(unsigned int startvec, · line 136

Quoted 10 lines
kernel/irq/affinity.c +2 -5
-             /* Spread the vectors per node */
-             vecs_per_node = (numvecs - (curvec - firstvec)) / nodes;
-
              /* Get the cpus on this node which are in the mask */
              cpumask_and(nmsk, cpu_mask, node_to_cpumask[n]);
-
-             /* Calculate the number of cpus per vector */
              ncpus = cpumask_weight(nmsk);
+             if (!ncpus)
+                     continue;

This shouldn't be possible, right? The nodemsk we're looping wouldn't have had that node set if no CPUs intersect the node_to_cpu_mask for that node, so the resulting cpumask should always have a non-zero weight.

     cpumask_and(nmsk, cpu_mask, node_to_cpumask[n]);

It is often true, see the following cases:

1) all CPUs in one node are not present

OR

2) all CPUs in one node are present

Quoted 9 lines

kernel/irq/affinity.c static int __irq_build_affinity_masks(unsigned int startvec, · line 168

Quoted 7 lines
kernel/irq/affinity.c +2 -0
@@ -153,16 +164,14 @@ static int __irq_build_affinity_masks(unsigned int startvec,
                      }
                      irq_spread_init_one(&masks[curvec].mask, nmsk,
                                              cpus_per_vec);
+                     if (++curvec >= last_affv)
+                             curvec = firstvec;

I'm not so sure about wrapping the vector to share it across nodes. We

Quoted 1 line

The wrapping is always there, not added by this patch.

We could avoid the wrapping completely given 'numvecs' > 'nodes', and it can be done by sorting each node's nr_cpus, will do it in V2.

Thanks,
Ming Lei

author
#5 lore raw

PATCH 2/2 genirq/affinity: spread vectors on node according to nr_cpu ratio

In reply to Ming Lei

Now __irq_build_affinity_masks() spreads vectors evenly per node, and all vectors may not be spread in case that each numa node has different CPU number, then the following warning in irq_build_affinity_masks() can be triggered:

	if (nr_present < numvecs)
		WARN_ON(nr_present + nr_others < numvecs);

Improve current spreading algorithm by assigning vectors according to the ratio of node's nr_cpu to nr_remaining_cpus.

Meantime the reported warning can be fixed.

Cc Christoph Hellwig <hch at lst.de>
Cc Keith Busch <kbusch at kernel.org>
Cc linux-nvme at lists.infradead.org,
Cc Jon Derrick <jonathan.derrick at intel.com>
Reported-by Jon Derrick <jonathan.derrick at intel.com>
Signed-off-by Ming Lei <ming.lei at redhat.com>
---
 kernel/irq/affinity.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
kernel/irq/affinity.c +17 -6
index bc3652a2c61b..76f3d1b27d00 100644
@@ -106,6 +106,7 @@ static int __irq_build_affinity_masks(unsigned int startvec,
 	unsigned int last_affv = firstvec + numvecs;
 	unsigned int curvec = startvec;
 	nodemask_t nodemsk = NODE_MASK_NONE;
+	unsigned remaining_cpus = 0;
 
 	if (!cpumask_weight(cpu_mask))
 		return 0;
@@ -126,6 +127,11 @@ static int __irq_build_affinity_masks(unsigned int startvec,
 		return numvecs;
 	}
 
+	for_each_node_mask(n, nodemsk) {
+		cpumask_and(nmsk, cpu_mask, node_to_cpumask[n]);
+		remaining_cpus += cpumask_weight(nmsk);
+	}
+
 	for_each_node_mask(n, nodemsk) {
 		unsigned int ncpus, v, vecs_to_assign, vecs_per_node;
 
@@ -135,17 +141,22 @@ static int __irq_build_affinity_masks(unsigned int startvec,
 		if (!ncpus)
 			continue;
 
+		if (remaining_cpus == 0)
+			break;
+
 		/*
 		 * Calculate the number of cpus per vector
 		 *
-		 * Spread the vectors evenly per node. If the requested
-		 * vector number has been reached, simply allocate one
-		 * vector for each remaining node so that all nodes can
-		 * be covered
+		 * Spread the vectors among CPUs on this node according
+		 * to the ratio of 'ncpus' to 'remaining_cpus'. If the
+		 * requested vector number has been reached, simply
+		 * spread one vector for each remaining node so that all
+		 * nodes can be covered
 		 */
 		if (numvecs > done)
 			vecs_per_node = max_t(unsigned,
-					(numvecs - done) / nodes, 1);
+					(numvecs - done) * ncpus /
+					remaining_cpus, 1);
 		else
 			vecs_per_node = 1;
 
@@ -169,7 +180,7 @@ static int __irq_build_affinity_masks(unsigned int startvec,
 		}
 
 		done += v;
-		--nodes;
+		remaining_cpus -= ncpus;
 	}
 	return done < numvecs ? done : numvecs;
 }
Signature · 1 line
2.20.1