Skip to contentSkip to conversation

rteval: kcompile: Tweak default source tarball selection

PATCH In review Valentin Schneider 2 participants 3 messages Apr 19, 2022 – May 3, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rteval: kcompile: Tweak default source tarball selection

Kcompile._WorkloadSetup() looks for a "tarball" and "tarfile" entry in the CfgSection, but I couldn't find a single setter for thoses. The only way for a user to specify a file is via --kcompile-source, which doesn't seem to be actually used by the module.

As it stands, this method will look for a tarball matching the hardcoded kernel_prefix.

Make Kcompile
- actually use --kcompile-source
- use the latest tarball in the loadsource if no source is specified.

---
I'm not entirely sure about that one, I tried to dig up the git history for uses of those "tarball" and "tarfile" options but didn't really get anywhere, so maybe I'm completely wrong...

---
 rteval/modules/loads/kcompile.py | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
rteval/modules/loads/kcompile.py +16 -6
index 367f8dc..2747913 100644
@@ -28,6 +28,7 @@ import sys
 import os
 import os.path
 import glob
+import re
 import subprocess
 from rteval.modules import rtevalRuntimeError
 from rteval.modules.loads import CommandLineLoad
@@ -35,7 +36,7 @@ from rteval.Log import Log
 from rteval.misc import expand_cpulist, compress_cpulist
 from rteval.systopology import SysTopology
 
-kernel_prefix = "linux-5.13"
+RE_KERNEL_TAR = re.compile(r"linux-(?P<maj>\d+)\.(?P<min>\d+).*\.tar.*")
 
 class KBuildJob:
     '''Class to manage a build job bound to a particular node'''
@@ -163,18 +164,27 @@ class Kcompile(CommandLineLoad):
             return
 
         # find our source tarball
-        if 'tarball' in self._cfg:
-            tarfile = os.path.join(self.srcdir, self._cfg.tarfile)
+        if self._cfg.source:
+            tarfile = os.path.join(self.srcdir, self._cfg.source)
             if not os.path.exists(tarfile):
                 raise rtevalRuntimeError(self, " tarfile %s does not exist!" % tarfile)
             self.source = tarfile
         else:
-            tarfiles = glob.glob(os.path.join(self.srcdir, "%s*" % kernel_prefix))
+            tarfiles = [f for f in os.listdir(self.srcdir)
+                        if RE_KERNEL_TAR.match(f)]
             if tarfiles:
-                self.source = tarfiles[0]
+                # Use (one of) the most recent kernels
+                def verkey(f):
+                    match = RE_KERNEL_TAR.match(f)
+                    return (match.group("maj"), match.group("min"))
+
+                tarfiles.sort(key=verkey)
+                self.source = tarfiles[-1]
             else:
                 raise rtevalRuntimeError(self, " no kernel tarballs found in %s" % self.srcdir)
 
+        kernel_prefix = re.search(r"linux-\d\.\d", self.source).group(0)
+
         # check for existing directory
         kdir = None
         names = os.listdir(self.builddir)
@@ -322,7 +332,7 @@ class Kcompile(CommandLineLoad):
 
 def ModuleParameters():
     return {"source":   {"descr": "Source tar ball",
-                         "default": "linux-5.13.2.tar.xz",
+                         "default": "",
                          "metavar": "TARBALL"},
             "jobspercore": {"descr": "Number of working threads per core",
                             "default": 2,
Signature · 1 line
2.27.0
#1 lore raw
Quoted 85 lines · Valentin Schneider

Kcompile._WorkloadSetup() looks for a "tarball" and "tarfile" entry in the CfgSection, but I couldn't find a single setter for thoses. The only way for a user to specify a file is via --kcompile-source, which doesn't seem to be actually used by the module.

As it stands, this method will look for a tarball matching the hardcoded kernel_prefix.

Make Kcompile
- actually use --kcompile-source
- use the latest tarball in the loadsource if no source is specified.

---
I'm not entirely sure about that one, I tried to dig up the git history for uses of those "tarball" and "tarfile" options but didn't really get anywhere, so maybe I'm completely wrong...

---
 rteval/modules/loads/kcompile.py | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
rteval/modules/loads/kcompile.py +16 -6
index 367f8dc..2747913 100644
@@ -28,6 +28,7 @@ import sys
 import os
 import os.path
 import glob
+import re
 import subprocess
 from rteval.modules import rtevalRuntimeError
 from rteval.modules.loads import CommandLineLoad
@@ -35,7 +36,7 @@ from rteval.Log import Log
 from rteval.misc import expand_cpulist, compress_cpulist
 from rteval.systopology import SysTopology
 
-kernel_prefix = "linux-5.13"
+RE_KERNEL_TAR = re.compile(r"linux-(?P<maj>\d+)\.(?P<min>\d+).*\.tar.*")
 
 class KBuildJob:
     '''Class to manage a build job bound to a particular node'''
@@ -163,18 +164,27 @@ class Kcompile(CommandLineLoad):
             return
 
         # find our source tarball
-        if 'tarball' in self._cfg:
-            tarfile = os.path.join(self.srcdir, self._cfg.tarfile)
+        if self._cfg.source:
+            tarfile = os.path.join(self.srcdir, self._cfg.source)
             if not os.path.exists(tarfile):
                 raise rtevalRuntimeError(self, " tarfile %s does not exist!" % tarfile)
             self.source = tarfile
         else:
-            tarfiles = glob.glob(os.path.join(self.srcdir, "%s*" % kernel_prefix))
+            tarfiles = [f for f in os.listdir(self.srcdir)
+                        if RE_KERNEL_TAR.match(f)]
             if tarfiles:
-                self.source = tarfiles[0]
+                # Use (one of) the most recent kernels
+                def verkey(f):
+                    match = RE_KERNEL_TAR.match(f)
+                    return (match.group("maj"), match.group("min"))
+
+                tarfiles.sort(key=verkey)
+                self.source = tarfiles[-1]
             else:
                 raise rtevalRuntimeError(self, " no kernel tarballs found in %s" % self.srcdir)
 
+        kernel_prefix = re.search(r"linux-\d\.\d", self.source).group(0)
+
         # check for existing directory
         kdir = None
         names = os.listdir(self.builddir)
@@ -322,7 +332,7 @@ class Kcompile(CommandLineLoad):
 
 def ModuleParameters():
     return {"source":   {"descr": "Source tar ball",
-                         "default": "linux-5.13.2.tar.xz",
+                         "default": "",
                          "metavar": "TARBALL"},
             "jobspercore": {"descr": "Number of working threads per core",
                             "default": 2,
Signature · 1 line
2.27.0

I think we wanted to make sure we were doing an apples to apples comparison, so we restricted the default kernel to the the one we supply in rteval-loads. I am thinking of changing this in the future to using the one supplied by the distribution, but still mulling this over.

However, --kcompile-source should work as advertised. Do you think you could resend a patch with just the part to make --kcompile-source work correctly?

Thanks

John

#2 lore raw
Quoted 24 lines · John Kacur
Quoted 12 lines · Valentin Schneider

Kcompile._WorkloadSetup() looks for a "tarball" and "tarfile" entry in the CfgSection, but I couldn't find a single setter for thoses. The only way for a user to specify a file is via --kcompile-source, which doesn't seem to be actually used by the module.

As it stands, this method will look for a tarball matching the hardcoded kernel_prefix.

Make Kcompile
- actually use --kcompile-source
- use the latest tarball in the loadsource if no source is specified.

Quoted 9 lines

---
I'm not entirely sure about that one, I tried to dig up the git history for uses of those "tarball" and "tarfile" options but didn't really get anywhere, so maybe I'm completely wrong...

---
 rteval/modules/loads/kcompile.py | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
Quoted 9 lines

I think we wanted to make sure we were doing an apples to apples comparison, so we restricted the default kernel to the the one we supply in rteval-loads. I am thinking of changing this in the future to using the one supplied by the distribution, but still mulling this over.

However, --kcompile-source should work as advertised. Do you think you could resend a patch with just the part to make --kcompile-source work correctly?

Sure thing, will do.

Quoted 3 lines

Thanks

John