Skip to contentSkip to conversation

rteval: Offline NUMA node bugfix

PATCH In review Valentin Schneider 2 participants 8 messages Apr 19, 2022 – May 3, 2022

lore
Share & tools
lore
#0 lore raw

cover rteval: Offline NUMA node bugfix

Hi folks,

I wanted to try out rteval on a NUMA system with one node offlined and got a few exceptions, here's my take on them. There's also a small cleanup as an added bonus.

Cheers,
Valentin

Valentin Schneider (3):
rteval: systopology: Fix offline NUMA node parsing
rteval: kcompile: Fix offline node handling
rteval: systopology: Slight CpuList.__expand_cpulist() cleanup

 rteval/modules/loads/kcompile.py |  5 ++++-
 rteval/systopology.py            | 15 +++++++++------
 2 files changed, 13 insertions(+), 7 deletions(-)

--
2.27.0

#1 lore raw

PATCH 1/3 rteval: systopology: Fix offline NUMA node parsing

An offline NUMA node will report in its cpulist an empty string. Unfortunately, "".split(sep=x) with x != None returns a list containing an empty string rather than an empty list, which causes CpuList._expand_cpulist() to try to run int(''), which ends up in the following exception:

ValueError: invalid literal for int() with base 10: ''

Prevent this by adding an early empty-string check.

---
 rteval/systopology.py | 4 ++++
 1 file changed, 4 insertions(+)
rteval/systopology.py +4 -0
index bf794ce..b2da7bb 100644
@@ -103,6 +103,10 @@ class CpuList:
         don't error check against online cpus
         """
         result = []
+
+        if not cpulist:
+            return result
+
         for part in cpulist.split(','):
             if '-' in part:
                 a, b = part.split('-')
Signature · 1 line
2.27.0
#2 lore raw

Re: [PATCH 1/3] rteval: systopology: Fix offline NUMA node parsing

Quoted 34 lines · Valentin Schneider

An offline NUMA node will report in its cpulist an empty string. Unfortunately, "".split(sep=x) with x != None returns a list containing an empty string rather than an empty list, which causes CpuList._expand_cpulist() to try to run int(''), which ends up in the following exception:

ValueError: invalid literal for int() with base 10: ''

Prevent this by adding an early empty-string check.

---
 rteval/systopology.py | 4 ++++
 1 file changed, 4 insertions(+)
rteval/systopology.py +4 -0
index bf794ce..b2da7bb 100644
@@ -103,6 +103,10 @@ class CpuList:
         don't error check against online cpus
         """
         result = []
+
+        if not cpulist:
+            return result
+
         for part in cpulist.split(','):
             if '-' in part:
                 a, b = part.split('-')
Signature · 1 line
2.27.0
#3 lore raw

PATCH 2/3 rteval: kcompile: Fix offline node handling

In reply to Valentin Schneider

Having an empty NumaNode but with CPUs attached to it (IOW they are all offline) causes kcompile.py to raise the following exception:

calc_jobs_per_cpu():
ratio = float(mem) / float(len(self.node))
ZeroDivisionError: float division by zero

Remove nodes that do have CPUs but none of which are online.

---
 rteval/modules/loads/kcompile.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
rteval/modules/loads/kcompile.py +4 -1
index 367f8dc..ac99964 100644
@@ -211,7 +211,10 @@ class Kcompile(CommandLineLoad):
 
         # remove nodes with no cpus available for running
         for node, cpus in self.cpus.items():
-            if not cpus:
+            # If the intersection between the node CPUs and the cpulist is empty
+            # then either the cpulist exludes that node, or the CPUs allowed by
+            # the cpulist are actually offline
+            if not set(self.topology.nodes[node].cpus.cpulist) & set(cpus):
                 self.nodes.remove(node)
                 self._log(Log.DEBUG, "node %s has no available cpus, removing" % node)
 
Signature · 1 line
2.27.0
#4 lore raw

Re: [PATCH 2/3] rteval: kcompile: Fix offline node handling

Quoted 34 lines · Valentin Schneider

Having an empty NumaNode but with CPUs attached to it (IOW they are all offline) causes kcompile.py to raise the following exception:

calc_jobs_per_cpu():
ratio = float(mem) / float(len(self.node))
ZeroDivisionError: float division by zero

Remove nodes that do have CPUs but none of which are online.

---
 rteval/modules/loads/kcompile.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
rteval/modules/loads/kcompile.py +4 -1
index 367f8dc..ac99964 100644
@@ -211,7 +211,10 @@ class Kcompile(CommandLineLoad):
 
         # remove nodes with no cpus available for running
         for node, cpus in self.cpus.items():
-            if not cpus:
+            # If the intersection between the node CPUs and the cpulist is empty
+            # then either the cpulist exludes that node, or the CPUs allowed by
+            # the cpulist are actually offline
+            if not set(self.topology.nodes[node].cpus.cpulist) & set(cpus):
                 self.nodes.remove(node)
                 self._log(Log.DEBUG, "node %s has no available cpus, removing" % node)
 
Signature · 1 line
2.27.0

Sorry, this isn't quite right.

The cpulist in kcompile is the list of cpus where the load modules will run. The user can specify it like this
--loads-cpulist=LIST

If the user does not specify a list (because they want it to run everywhere) then the cpulist is empty. Your patch was working for you because the cpulist was empty, but that has nothing to do with whether the cpu is online or not.

systopology will fetch a list of cpus and consider whether they are online or not. So, I think the solution is to delete the method in kcompile and just use the one in systopology.

Sending another mail with the patch.

Thanks

John Kacur

#5 lore raw

PATCH 3/3 rteval: systopology: Slight CpuList.__expand_cpulist() cleanup

In reply to Valentin Schneider

This method currently aggregates CPUs into a list, then converts this to set and then back to list. The aggregation can instead be done directly into a set.

(as an offside, it would make more sense for CpuList to have its storage be a set in the first place as duplicate CPU ids don't make sense for it, but that's a separate discussion :-))

The integer conversion of the "a-b" pattern can also be condensed into a single map() expression.

---
 rteval/systopology.py | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
rteval/systopology.py +7 -8
index b2da7bb..2a28f9c 100644
@@ -102,20 +102,19 @@ class CpuList:
         """ expand a range string into an array of cpu numbers
         don't error check against online cpus
         """
-        result = []
-
         if not cpulist:
-            return result
+            return []
+
+        result = set()
 
         for part in cpulist.split(','):
             if '-' in part:
-                a, b = part.split('-')
-                a, b = int(a), int(b)
-                result.extend(list(range(a, b + 1)))
+                a, b = map(int, part.split('-'))
+                result |= set(range(a, b + 1))
             else:
                 a = int(part)
-                result.append(a)
-        return [int(i) for i in list(set(result))]
+                result |= {a}
+        return list(result)
 
     def getcpulist(self):
         """ return the list of cpus tracked """
Signature · 1 line
2.27.0
#6 lore raw

Re: [PATCH 3/3] rteval: systopology: Slight CpuList.__expand_cpulist() cleanup

Quoted 52 lines · Valentin Schneider

This method currently aggregates CPUs into a list, then converts this to set and then back to list. The aggregation can instead be done directly into a set.

(as an offside, it would make more sense for CpuList to have its storage be a set in the first place as duplicate CPU ids don't make sense for it, but that's a separate discussion :-))

The integer conversion of the "a-b" pattern can also be condensed into a single map() expression.

---
 rteval/systopology.py | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
rteval/systopology.py +7 -8
index b2da7bb..2a28f9c 100644
@@ -102,20 +102,19 @@ class CpuList:
         """ expand a range string into an array of cpu numbers
         don't error check against online cpus
         """
-        result = []
-
         if not cpulist:
-            return result
+            return []
+
+        result = set()
 
         for part in cpulist.split(','):
             if '-' in part:
-                a, b = part.split('-')
-                a, b = int(a), int(b)
-                result.extend(list(range(a, b + 1)))
+                a, b = map(int, part.split('-'))
+                result |= set(range(a, b + 1))
             else:
                 a = int(part)
-                result.append(a)
-        return [int(i) for i in list(set(result))]
+                result |= {a}
+        return list(result)
 
     def getcpulist(self):
         """ return the list of cpus tracked """
Signature · 1 line
2.27.0

I'm guessing that the reason for the "set" was to remove any potential duplicates. Duplicates are not normally a problem in rteval, but if you want to ensure you handle any user input correctly, it makes sense to do that. I think your code is correct, but I'm not sure if it buys us anything.

John

#7 lore raw

Re: [PATCH 3/3] rteval: systopology: Slight CpuList.__expand_cpulist() cleanup

Quoted 48 lines · John Kacur
Quoted 46 lines · Valentin Schneider

This method currently aggregates CPUs into a list, then converts this to set and then back to list. The aggregation can instead be done directly into a set.

(as an offside, it would make more sense for CpuList to have its storage be a set in the first place as duplicate CPU ids don't make sense for it, but that's a separate discussion :-))

The integer conversion of the "a-b" pattern can also be condensed into a single map() expression.

---
 rteval/systopology.py | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)
rteval/systopology.py +7 -8
index b2da7bb..2a28f9c 100644
@@ -102,20 +102,19 @@ class CpuList:
         """ expand a range string into an array of cpu numbers
         don't error check against online cpus
         """
-        result = []
-
         if not cpulist:
-            return result
+            return []
+
+        result = set()
 
         for part in cpulist.split(','):
             if '-' in part:
-                a, b = part.split('-')
-                a, b = int(a), int(b)
-                result.extend(list(range(a, b + 1)))
+                a, b = map(int, part.split('-'))
+                result |= set(range(a, b + 1))
             else:
                 a = int(part)
-                result.append(a)
-        return [int(i) for i in list(set(result))]
+                result |= {a}
+        return list(result)
 
Quoted 13 lines
Quoted 6 lines
rteval/systopology.py +0 -1
     def getcpulist(self):
         """ return the list of cpus tracked """
--

2.27.0

I'm guessing that the reason for the "set" was to remove any potential duplicates. Duplicates are not normally a problem in rteval, but if you want to ensure you handle any user input correctly, it makes sense to do that. I think your code is correct, but I'm not sure if it buys us anything.

It doesn't really, it should be functionaly identical to the existing code
- it's just "drive-by cleanup" really :)

Quoted 1 line

John