rteval: Add cgroup v2 cpuset integration for dynamic CPU isolation
cover rteval: Add cgroup v2 cpuset integration for dynamic CPU isolation
This patch series adds cpuset integration to rteval, providing dynamic CPU isolation using cgroup v2 cpusets as an alternative or complement to the traditional isolcpus boot parameter.
Key features:
- New --cpusets flag enables cpuset-based isolation for measurements
- --housekeeping flag now works without requiring isolcpus when used with --cpusets
- --cleanup-cpusets utility flag removes leftover cpusets from crashed runs
- Automatic cleanup using Python context managers
- Full backward compatibility (no cpusets created without --cpusets flag)
Implementation philosophy:
The simplified implementation mirrors rteval's existing isolcpus behavior:
- Measurement threads get cpuset isolation (quiet CPUs for latency testing)
- Housekeeping tasks optionally get cpuset isolation (keep system noise away)
- Load processes use taskset affinity (no cpuset needed, just like without isolcpus)
This approach avoids complexity of migrating short-lived load processes while providing the isolation needed for accurate latency measurements.
Benefits over isolcpus alone:
- No reboot required to change CPU isolation
- Dynamic reconfiguration for testing different scenarios
- Housekeeping can use any CPUs (not restricted to non-isolated CPUs)
- Can be combined with isolcpus for layered isolation
The patch series includes:
1. Core cpuset infrastructure and manager
2. Cleanup utility for leftover cpusets
3. Command-line argument reorganization for clarity
4. Simplified implementation (measurements + housekeeping only)
5. Development directory for untracked test files
6. Documentation updates
7. Warning for edge case (all CPUs isolated)
Tested on systems with and without isolcpus, with comprehensive test suite validating all scenarios including cleanup on interrupt (Ctrl+C).
John Kacur (7):
rteval: Add cpuset integration infrastructure (WIP)
rteval: Add --cleanup-cpusets flag to remove leftover cpusets
rteval: Reorganize command-line arguments into logical groups
rteval: Simplify cpuset implementation to match isolcpus philosophy
rteval: Add local/ directory to .gitignore for development files
rteval: Update cpuset documentation to reflect simplified
implementation
rteval: Warn when all CPUs isolated and loads have no CPUs
.gitignore | 1 + rteval-cmd | 193 +++++++++++++------ rteval/__init__.py | 12 +- rteval/cpusetmanager.py | 224 +++++++++++++++++++++++ rteval/modules/__init__.py | 30 +++ rteval/modules/loads/hackbench.py | 1 - rteval/modules/loads/kcompile.py | 1 - rteval/modules/measurement/cyclictest.py | 5 + rteval/modules/measurement/timerlat.py | 6 + 9 files changed, 419 insertions(+), 54 deletions(-) create mode 100644 rteval/cpusetmanager.py
Signature · 1 line
2.54.0
PATCH 1/7 rteval: Add cpuset integration infrastructure (WIP)
Add --cpusets flag to isolate workloads using cgroup v2 cpusets. This provides an alternative to isolcpus kernel parameter, allowing dynamic CPU isolation without requiring boot parameter changes.
This commit includes the complete cpuset infrastructure:
- CpusetManager class for cpuset lifecycle management
- get_subprocess_pids() methods in all load/measurement modules
- Integration in rteval-cmd and RtEval class
- Smart cpuset creation (1-3 cpusets based on CPU configuration)
- Startup cleanup of leftover cpusets from previous runs
- Comprehensive test suite (test-cpusets.sh) with 7 passing tests
Current status:
- Cpuset creation: Working
- Housekeeping migration: Working
- Startup cleanup: Working
- All 7 tests passing
Known issue: Child processes spawned by loads don't inherit cpuset because parent is migrated after children spawn. This will be fixed in a follow-up commit by migrating processes before they spawn children.
Changes:
- rteval-cmd: Add --cpusets flag and startup cleanup call
- rteval/__init__.py: Accept cpuset_manager, migrate after Unleash()
- rteval/cpusetmanager.py: New cpuset manager with cleanup
- rteval/modules/__init__.py: Add get_subprocess_pids() base method
- rteval/modules/loads/*.py: Implement get_subprocess_pids()
- rteval/modules/measurement/*.py: Implement get_subprocess_pids()
- CPUSET-INTEGRATION-PLAN.md: Complete implementation plan
- test-cpusets.sh: Automated test suite
--- rteval-cmd | 85 ++++++- rteval/__init__.py | 17 +- rteval/cpusetmanager.py | 304 +++++++++++++++++++++++ rteval/modules/__init__.py | 30 +++ rteval/modules/loads/hackbench.py | 7 + rteval/modules/loads/kcompile.py | 7 + rteval/modules/loads/stressng.py | 6 + rteval/modules/measurement/cyclictest.py | 5 + rteval/modules/measurement/timerlat.py | 6 + 9 files changed, 454 insertions(+), 13 deletions(-) create mode 100644 rteval/cpusetmanager.py
rteval-cmd +73 -12
@@ -119,6 +119,9 @@ def parse_options(cfg, parser, cmdargs): parser.add_argument("--warn-non-isolated-core-sharing", dest="rteval___warn_non_isolated_core_sharing", action="store_true", default=False, help="warn about measurement and load CPUs sharing cores even when neither is isolated") + parser.add_argument("--cpusets", dest="rteval___cpusets", + action="store_true", default=False, + help="use cgroup v2 cpusets to isolate measurement, load, and housekeeping workloads") parser.add_argument("-s", "--sysreport", dest="rteval___sysreport", action="store_true", default=rtevcfg.sysreport, help=f'run sysreport to collect system data (default: {rtevcfg.sysreport})')
@@ -380,8 +383,15 @@ if __name__ == '__main__': # Validate and process housekeeping CPUs housekeeping_cpus = [] if rtevcfg.housekeeping: - housekeeping_cpus = validate_housekeeping_cpus(rtevcfg.housekeeping) - logger.log(Log.DEBUG, f"housekeeping cpulist: {collapse_cpulist(housekeeping_cpus)}") + if rtevcfg.cpusets: + # With cpusets, housekeeping doesn't require isolcpus + # Just parse and validate it's a valid CPU list + housekeeping_cpus = CpuList(rtevcfg.housekeeping).online().cpus + logger.log(Log.DEBUG, f"housekeeping cpulist: {collapse_cpulist(housekeeping_cpus)}") + else: + # Without cpusets, require housekeeping CPUs to be in isolcpus + housekeeping_cpus = validate_housekeeping_cpus(rtevcfg.housekeeping) + logger.log(Log.DEBUG, f"housekeeping cpulist: {collapse_cpulist(housekeeping_cpus)}") # Remember if cpulists were explicitly set by the user before running # parse_cpulist_from_config, which generates default value for them
@@ -491,7 +501,31 @@ if __name__ == '__main__': cpupower_controller = cpupower.Cpupower(msrcfg.cpulist, msrcfg.idlestate, logger=logger) cpupower_controller.enable_idle_state() - rteval = RtEval(config, loadmods, measuremods, logger) + # Cpuset setup if requested + cpuset_manager = None + if rtevcfg.cpusets: + from rteval.cpusetmanager import CpusetManager + from rteval.cpuset import CpusetsInit + + # Check if cpusets are supported + cpusets_init = CpusetsInit() + if not cpusets_init.supported: + logger.log(Log.ERR, "Cpusets requested but cgroup v2 cpuset controller not available") + sys.exit(1) + + # Clean up any leftover cpusets from previous runs + CpusetManager.cleanup_leftover_cpusets(logger) + + # Create manager (will create cpusets in __enter__) + cpuset_manager = CpusetManager( + housekeeping_cpus=housekeeping_cpus, + measurement_cpus=msrcfg_cpus, + load_cpus=ldcfg_cpus, + logger=logger + ) + + # Pass cpuset_manager to RtEval (None if not using cpusets) + rteval = RtEval(config, loadmods, measuremods, logger, cpuset_manager) # Add core sharing warnings to the XML report now that CPU lists are finalized rteval._sysinfo.add_core_sharing_warnings(housekeeping_cpus, msrcfg_cpus, ldcfg_cpus,
@@ -502,17 +536,44 @@ if __name__ == '__main__': if rtevcfg.onlyload: # If --onlyload were given, just kick off the loads and nothing more # No reports will be created. - loadmods.Start() - nthreads = loadmods.Unleash() - logger.log(Log.INFO, f"Started {nthreads} load threads - will run for {rtevcfg.duration} seconds") - logger.log(Log.INFO, "No measurements will be performed, due to the --onlyload option") - time.sleep(rtevcfg.duration) - loadmods.Stop() - ec = 0 + if rtevcfg.cpusets: + with cpuset_manager: + cpuset_manager.migrate_root_tasks_to_housekeeping() + + loadmods.Start() + nthreads = loadmods.Unleash() + + # Migrate load processes + time.sleep(0.5) # Give processes time to spawn + load_pids = loadmods.GetSubprocessPids() + cpuset_manager.migrate_load_threads(load_pids) + + logger.log(Log.INFO, f"Started {nthreads} load threads - will run for {rtevcfg.duration} seconds") + logger.log(Log.INFO, "No measurements will be performed, due to the --onlyload option") + time.sleep(rtevcfg.duration) + loadmods.Stop() + ec = 0 + else: + loadmods.Start() + nthreads = loadmods.Unleash() + logger.log(Log.INFO, f"Started {nthreads} load threads - will run for {rtevcfg.duration} seconds") + logger.log(Log.INFO, "No measurements will be performed, due to the --onlyload option") + time.sleep(rtevcfg.duration) + loadmods.Stop() + ec = 0 else: # ... otherwise, run the full measurement suite with loads - ec = rteval.Measure() - logger.log(Log.DEBUG, f"exiting with exit code: {ec}") + if rtevcfg.cpusets: + with cpuset_manager: + # Migrate existing tasks to housekeeping (if housekeeping cpuset exists) + cpuset_manager.migrate_root_tasks_to_housekeeping() + + # RtEval will handle migrating measurement/load processes internally + ec = rteval.Measure() + logger.log(Log.DEBUG, f"exiting with exit code: {ec}") + else: + ec = rteval.Measure() + logger.log(Log.DEBUG, f"exiting with exit code: {ec}") # restore previous idle state settings if msrcfg.idlestate:
rteval/__init__.py +16 -1
@@ -49,7 +49,7 @@ def except_hook(args): stopsig.set() class RtEval(rtevalReport): - def __init__(self, config, loadmods, measuremods, logger): + def __init__(self, config, loadmods, measuremods, logger, cpuset_manager=None): self.__version = RTEVAL_VERSION threading.excepthook = except_hook
@@ -66,6 +66,7 @@ class RtEval(rtevalReport): self.__logger = logger self._loadmods = loadmods self._measuremods = measuremods + self._cpuset_manager = cpuset_manager self.__rtevcfg = self.__cfg.GetSection('rteval') self.__reportdir = None
@@ -184,6 +185,20 @@ class RtEval(rtevalReport): self._measuremods.Unleash() measure_start = datetime.now() + # After Unleash(), do cpuset migration + if self._cpuset_manager: + # Small delay to let processes spawn + time.sleep(0.5) + + if self._loadmods: + load_pids = self._loadmods.GetSubprocessPids() + self._cpuset_manager.migrate_load_threads(load_pids) + + meas_pids = self._measuremods.GetSubprocessPids() + self._cpuset_manager.migrate_measurement_threads(meas_pids) + + self.__logger.log(Log.INFO, "Processes migrated to cpusets") + # wait for time to expire or thread to die signal.signal(signal.SIGINT, sig_handler) signal.signal(signal.SIGTERM, sig_handler)
rteval/cpusetmanager.py new file +304 -0
304 lines not on this page · raw message
rteval/modules/__init__.py +30 -0
@@ -175,6 +175,19 @@ class rtevalModulePrototype(threading.Thread): return self._donotrun is False + def get_subprocess_pids(self): + """ + Return list of subprocess PIDs spawned by this module + + This method should be overridden by modules that spawn subprocesses. + Default implementation returns an empty list. + + Returns: + list: List of integer PIDs of spawned subprocesses + """ + return [] + + def __run(self): "Workload thread runner - takes care of keeping the workload running as long as needed" if self.shouldStop():
@@ -539,6 +552,23 @@ class RtEvalModules: self._logger.log(Log.DEBUG, f"All {self._module_type} modules completed") + def GetSubprocessPids(self): + """ + Get subprocess PIDs from all loaded modules + + Collects PIDs from all modules that will run by calling each module's + get_subprocess_pids() method. + + Returns: + list: List of integer PIDs from all modules + """ + pids = [] + for (modname, mod) in self.__modules: + if mod.WorkloadWillRun(): + pids.extend(mod.get_subprocess_pids()) + return pids + + def MakeReport(self): """Collects all the loaded modules reports in a single libxml2.xmlNode() object"""
rteval/modules/loads/hackbench.py +7 -0
@@ -159,6 +159,13 @@ class Hackbench(CommandLineLoad): # As hackbench is short-lived, lets pretend it is always alive return True + def get_subprocess_pids(self): + """Return PIDs of all running hackbench processes""" + pids = [] + for task in self.tasks.values(): + if task and task.poll() is None: + pids.append(task.pid) + return pids def _WorkloadCleanup(self): if self._donotrun:
rteval/modules/loads/kcompile.py +7 -0
@@ -314,6 +314,13 @@ class Kcompile(CommandLineLoad): return True + def get_subprocess_pids(self): + """Return PIDs of all kcompile build jobs""" + pids = [] + for job in self.buildjobs.values(): + if job.jobid and job.isrunning(): + pids.append(job.jobid.pid) + return pids def _WorkloadCleanup(self): if self._donotrun:
rteval/modules/loads/stressng.py +6 -0
@@ -110,6 +110,12 @@ class Stressng(CommandLineLoad): return self.process.poll() is None return False + def get_subprocess_pids(self): + """Return PID of stress-ng process""" + if self.started and self.process and self.process.poll() is None: + return [self.process.pid] + return [] + def _WorkloadCleanup(self): " Makesure to kill stress-ng before rteval ends " if not self.started:
rteval/modules/measurement/cyclictest.py +5 -0
@@ -296,6 +296,11 @@ class Cyclictest(rtevalModulePrototype): return self.__cyclicprocess.poll() is None return False + def get_subprocess_pids(self): + """Return PID of cyclictest process""" + if self.__cyclicprocess and self.__cyclicprocess.poll() is None: + return [self.__cyclicprocess.pid] + return [] def _parse_max_latencies(self, line): if not line.startswith('# Max Latencies: '):
rteval/modules/measurement/timerlat.py +6 -0
@@ -273,6 +273,12 @@ class Timerlat(rtevalModulePrototype): return self.__timerlat_process.poll() is None return False + def get_subprocess_pids(self): + """Return PID of timerlat process""" + if self.__timerlat_process and self.__timerlat_process.poll() is None: + return [self.__timerlat_process.pid] + return [] + def _WorkloadCleanup(self): if not self.__started: return
Signature · 1 line
2.54.0
PATCH 2/7 rteval: Add --cleanup-cpusets flag to remove leftover cpusets
In reply to John KacurAdd a new command-line flag --cleanup-cpusets that removes any leftover rteval cpusets from previous runs and exits. This is useful when rteval was killed and didn't clean up properly, or when users want to manually clean up without starting a new rteval run.
The cleanup function already existed and runs automatically at startup when using --cpusets, but this flag makes it available as a standalone utility command.
Changes:
- rteval-cmd: Add --cleanup-cpusets argument and early handler
- rteval/cpusetmanager.py: Change log level to INFO for "no cpusets found" message so users get feedback when running --cleanup-cpusets
Example usage:
sudo ./rteval-cmd --cleanup-cpusets
--- rteval-cmd | 11 +++++++++++ rteval/cpusetmanager.py | 2 +- 2 files changed, 12 insertions(+), 1 deletion(-)
rteval-cmd +11 -0
@@ -122,6 +122,9 @@ def parse_options(cfg, parser, cmdargs): parser.add_argument("--cpusets", dest="rteval___cpusets", action="store_true", default=False, help="use cgroup v2 cpusets to isolate measurement, load, and housekeeping workloads") + parser.add_argument("--cleanup-cpusets", dest="rteval___cleanup_cpusets", + action="store_true", default=False, + help="remove any leftover rteval cpusets from previous runs and exit") parser.add_argument("-s", "--sysreport", dest="rteval___sysreport", action="store_true", default=rtevcfg.sysreport, help=f'run sysreport to collect system data (default: {rtevcfg.sysreport})')
@@ -167,6 +170,14 @@ def parse_options(cfg, parser, cmdargs): print(f"rteval version {RTEVAL_VERSION}") sys.exit(0) + if cmd_opts.rteval___cleanup_cpusets: + from rteval.cpusetmanager import CpusetManager + from rteval.Log import Log + logger = Log() + logger.SetLogVerbosity(Log.INFO) + CpusetManager.cleanup_leftover_cpusets(logger) + sys.exit(0) + if cmd_opts.rteval___duration: mult = 1.0 v = cmd_opts.rteval___duration.lower()
rteval/cpusetmanager.py +1 -1
@@ -41,7 +41,7 @@ class CpusetManager: """ cpuset_dirs = glob.glob('/sys/fs/cgroup/rteval_*/') if not cpuset_dirs: - logger.log(Log.DEBUG, "No leftover rteval cpusets found") + logger.log(Log.INFO, "No leftover rteval cpusets found") return logger.log(Log.INFO, f"Cleaning up {len(cpuset_dirs)} leftover rteval cpusets from previous run")
Signature · 1 line
2.54.0
PATCH 3/7 rteval: Reorganize command-line arguments into logical groups
In reply to John KacurReorganize rteval-cmd arguments to improve code maintainability and readability:
- Group arguments by function: Utility/Control, Execution Mode, CPU Configuration, Configuration & Path, Logging & Verbosity, Reporting, and Advanced/Special options
- Alphabetize arguments within each group by long option name
- Keep help output clean without category headers (users still see simple alphabetical list)
This reorganization makes the code easier to navigate and maintain while keeping the user-facing help output simple and uncluttered.
--- rteval-cmd | 104 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 59 insertions(+), 45 deletions(-)
rteval-cmd +59 -45
@@ -95,69 +95,83 @@ def parse_options(cfg, parser, cmdargs): # thus they are prefixed with 'rteval___'. # See rteval/rtevalConfig::UpdateFromOptionParser() method for more info # + + # Utility/Control Options + parser.add_argument("--cleanup-cpusets", dest="rteval___cleanup_cpusets", + action="store_true", default=False, + help="remove any leftover rteval cpusets from previous runs and exit") + parser.add_argument("-V", "--version", dest="rteval___version", + action='store_true', default=False, + help='print rteval version and exit') + + # Execution Mode Options parser.add_argument("-d", "--duration", dest="rteval___duration", type=str, default=rtevcfg.duration, metavar="DURATION", help=f"specify length of test run (default: {rtevcfg.duration})") - parser.add_argument("-v", "--verbose", dest="rteval___verbose", - action="store_true", default=rtevcfg.verbose, - help=f"turn on verbose prints (default: {rtevcfg.verbose})") - parser.add_argument("-q", "--quiet", dest="rteval___quiet", - action="store_true", default=rtevcfg.quiet, - help=f"turn on quiet mode (default: {rtevcfg.quiet})") - parser.add_argument("-w", "--workdir", dest="rteval___workdir", - type=str, default=rtevcfg.workdir, metavar="DIRECTORY", - help=f"top directory for rteval data (default: {rtevcfg.workdir})") - parser.add_argument("-l", "--loaddir", dest="rteval___srcdir", - type=str, default=rtevcfg.srcdir, metavar="DIRECTORY", - help=f"directory for load source tarballs (default: {rtevcfg.srcdir})") - parser.add_argument("-i", "--installdir", dest="rteval___installdir", - type=str, default=rtevcfg.installdir, metavar="DIRECTORY", - help=f"place to locate installed templates (default: {rtevcfg.installdir})") + parser.add_argument("--noload", dest="rteval___noload", + action="store_true", default=False, + help="only run the measurements (don't run loads)") + parser.add_argument("-O", "--onlyload", dest="rteval___onlyload", + action='store_true', default=False, + help="only run the loads (don't run measurement threads)") + + # CPU Configuration Options + parser.add_argument("--cpusets", dest="rteval___cpusets", + action="store_true", default=False, + help="use cgroup v2 cpusets to isolate measurement, load, and housekeeping workloads") parser.add_argument("--housekeeping", dest="rteval___housekeeping", type=str, default="", metavar="CPULIST", help="isolated CPUs reserved for system tasks (not used by rteval)") parser.add_argument("--warn-non-isolated-core-sharing", dest="rteval___warn_non_isolated_core_sharing", action="store_true", default=False, help="warn about measurement and load CPUs sharing cores even when neither is isolated") - parser.add_argument("--cpusets", dest="rteval___cpusets", - action="store_true", default=False, - help="use cgroup v2 cpusets to isolate measurement, load, and housekeeping workloads") - parser.add_argument("--cleanup-cpusets", dest="rteval___cleanup_cpusets", - action="store_true", default=False, - help="remove any leftover rteval cpusets from previous runs and exit") - parser.add_argument("-s", "--sysreport", dest="rteval___sysreport", - action="store_true", default=rtevcfg.sysreport, - help=f'run sysreport to collect system data (default: {rtevcfg.sysreport})') - parser.add_argument("-D", '--debug', dest='rteval___debugging', - action='store_true', default=rtevcfg.debugging, - help=f'turn on debug prints (default: {rtevcfg.debugging})') - parser.add_argument("-Z", '--summarize', dest='rteval___summarize', - nargs='+', default=None, metavar='XMLFILE', - help='summarize one or more already existing XML reports') - parser.add_argument("-H", '--raw-histogram', dest='rteval___rawhistogram', - nargs='+', default=None, metavar='XMLFILE', - help='Generate raw histogram data for one or more already existing XML reports') + + # Configuration & Path Options parser.add_argument("-f", "--inifile", dest="rteval___inifile", type=str, default=None, metavar="FILE", help="initialization file for configuring loads and behavior") - parser.add_argument("-a", "--annotate", dest="rteval___annotate", - type=str, default=None, metavar="STRING", - help="Add a little annotation which is stored in the report") + parser.add_argument("-i", "--installdir", dest="rteval___installdir", + type=str, default=rtevcfg.installdir, metavar="DIRECTORY", + help=f"place to locate installed templates (default: {rtevcfg.installdir})") + parser.add_argument("-l", "--loaddir", dest="rteval___srcdir", + type=str, default=rtevcfg.srcdir, metavar="DIRECTORY", + help=f"directory for load source tarballs (default: {rtevcfg.srcdir})") + parser.add_argument("-w", "--workdir", dest="rteval___workdir", + type=str, default=rtevcfg.workdir, metavar="DIRECTORY", + help=f"top directory for rteval data (default: {rtevcfg.workdir})") + + # Logging & Verbosity Options + parser.add_argument("-D", '--debug', dest='rteval___debugging', + action='store_true', default=rtevcfg.debugging, + help=f'turn on debug prints (default: {rtevcfg.debugging})') parser.add_argument("-L", "--logging", dest="rteval___logging", action='store_true', default=False, help='log the output of the loads in the report directory') - parser.add_argument("-O", "--onlyload", dest="rteval___onlyload", - action='store_true', default=False, - help="only run the loads (don't run measurement threads)") - parser.add_argument("-V", "--version", dest="rteval___version", - action='store_true', default=False, - help='print rteval version and exit') + parser.add_argument("-q", "--quiet", dest="rteval___quiet", + action="store_true", default=rtevcfg.quiet, + help=f"turn on quiet mode (default: {rtevcfg.quiet})") + parser.add_argument("-v", "--verbose", dest="rteval___verbose", + action="store_true", default=rtevcfg.verbose, + help=f"turn on verbose prints (default: {rtevcfg.verbose})") + + # Reporting Options + parser.add_argument("-a", "--annotate", dest="rteval___annotate", + type=str, default=None, metavar="STRING", + help="Add a little annotation which is stored in the report") + parser.add_argument("-H", '--raw-histogram', dest='rteval___rawhistogram', + nargs='+', default=None, metavar='XMLFILE', + help='Generate raw histogram data for one or more already existing XML reports') + parser.add_argument("-Z", '--summarize', dest='rteval___summarize', + nargs='+', default=None, metavar='XMLFILE', + help='summarize one or more already existing XML reports') + parser.add_argument("-s", "--sysreport", dest="rteval___sysreport", + action="store_true", default=rtevcfg.sysreport, + help=f'run sysreport to collect system data (default: {rtevcfg.sysreport})') + + # Advanced/Special Options parser.add_argument("-S", "--source-download", nargs="?", dest="rteval___srcdownload", type=str, default=None, const='', metavar="KERNEL_VERSION", help='download a source kernel from kernel.org and exit') - parser.add_argument("--noload", dest="rteval___noload", - action="store_true", default=False, - help="only run the measurements (don't run loads)") cmd_opts = parser.parse_args(args=cmdargs)
Signature · 1 line
2.54.0
PATCH 4/7 rteval: Simplify cpuset implementation to match isolcpus philosophy
In reply to John KacurSimplify the cpuset integration to create only 1-2 cpusets instead of 1-3, matching how rteval works with isolcpus: measurements get isolation, loads use affinity constraints only.
Changes:
- rteval_housekeeping: Created only if --housekeeping specified (system tasks)
- rteval_measurement: Always created for measurement threads
- Loads: Use taskset for CPU affinity (no cpuset migration)
This approach:
- Mirrors isolcpus behavior (measurements isolated, loads constrained via affinity)
- Eliminates complexity of load cpuset migration
- Removes race conditions with short-lived load processes
- Simplifies codebase while maintaining CPU isolation goals
Modified files:
- rteval/cpusetmanager.py: Removed load_cpus parameter and migrate_load_threads()
- rteval-cmd: Updated CpusetManager instantiation, removed load migration
- rteval/__init__.py: Simplified to only migrate measurement threads
- rteval/modules/loads/*: Removed get_subprocess_pids() methods (no longer needed)
--- rteval-cmd | 8 +- rteval/__init__.py | 11 +-- rteval/cpusetmanager.py | 131 ++++++------------------------ rteval/modules/loads/hackbench.py | 8 -- rteval/modules/loads/kcompile.py | 8 -- rteval/modules/loads/stressng.py | 6 -- 6 files changed, 30 insertions(+), 142 deletions(-)
rteval-cmd +2 -6
@@ -542,10 +542,10 @@ if __name__ == '__main__': CpusetManager.cleanup_leftover_cpusets(logger) # Create manager (will create cpusets in __enter__) + # Note: Load CPUs not needed - loads use taskset for CPU affinity cpuset_manager = CpusetManager( housekeeping_cpus=housekeeping_cpus, measurement_cpus=msrcfg_cpus, - load_cpus=ldcfg_cpus, logger=logger )
@@ -568,11 +568,7 @@ if __name__ == '__main__': loadmods.Start() nthreads = loadmods.Unleash() - # Migrate load processes - time.sleep(0.5) # Give processes time to spawn - load_pids = loadmods.GetSubprocessPids() - cpuset_manager.migrate_load_threads(load_pids) - + # Loads use taskset for CPU affinity (no cpuset migration needed) logger.log(Log.INFO, f"Started {nthreads} load threads - will run for {rtevcfg.duration} seconds") logger.log(Log.INFO, "No measurements will be performed, due to the --onlyload option") time.sleep(rtevcfg.duration)
rteval/__init__.py +3 -8
@@ -185,20 +185,15 @@ class RtEval(rtevalReport): self._measuremods.Unleash() measure_start = datetime.now() - # After Unleash(), do cpuset migration + # After Unleash(), migrate measurement threads to cpuset + # Note: Load threads use taskset for CPU affinity (no cpuset migration needed) if self._cpuset_manager: - # Small delay to let processes spawn + # Small delay to let measurement processes spawn time.sleep(0.5) - if self._loadmods: - load_pids = self._loadmods.GetSubprocessPids() - self._cpuset_manager.migrate_load_threads(load_pids) - meas_pids = self._measuremods.GetSubprocessPids() self._cpuset_manager.migrate_measurement_threads(meas_pids) - self.__logger.log(Log.INFO, "Processes migrated to cpusets") - # wait for time to expire or thread to die signal.signal(signal.SIGINT, sig_handler) signal.signal(signal.SIGTERM, sig_handler)
rteval/cpusetmanager.py +25 -106
200 lines not on this page · raw message
rteval/modules/loads/hackbench.py +0 -8
@@ -159,14 +159,6 @@ class Hackbench(CommandLineLoad): # As hackbench is short-lived, lets pretend it is always alive return True - def get_subprocess_pids(self): - """Return PIDs of all running hackbench processes""" - pids = [] - for task in self.tasks.values(): - if task and task.poll() is None: - pids.append(task.pid) - return pids - def _WorkloadCleanup(self): if self._donotrun: return
rteval/modules/loads/kcompile.py +0 -8
@@ -314,14 +314,6 @@ class Kcompile(CommandLineLoad): return True - def get_subprocess_pids(self): - """Return PIDs of all kcompile build jobs""" - pids = [] - for job in self.buildjobs.values(): - if job.jobid and job.isrunning(): - pids.append(job.jobid.pid) - return pids - def _WorkloadCleanup(self): if self._donotrun: return
rteval/modules/loads/stressng.py +0 -6
@@ -110,12 +110,6 @@ class Stressng(CommandLineLoad): return self.process.poll() is None return False - def get_subprocess_pids(self): - """Return PID of stress-ng process""" - if self.started and self.process and self.process.poll() is None: - return [self.process.pid] - return [] - def _WorkloadCleanup(self): " Makesure to kill stress-ng before rteval ends " if not self.started:
Signature · 1 line
2.54.0
PATCH 5/7 rteval: Add local/ directory to .gitignore for development files
In reply to John KacurAdd local/ directory to hold development scripts, notes, and other files that should not be tracked in git.
--- .gitignore | 1 + 1 file changed, 1 insertion(+)
.gitignore +1 -0
@@ -28,3 +28,4 @@ MANIFEST stamp-h1 autom4te.cache/ rteval-[0-9]* +local/
Signature · 1 line
2.54.0
PATCH 6/7 rteval: Update cpuset documentation to reflect simplified implementation
In reply to John KacurUpdate help text and docstrings to clarify that the simplified cpuset implementation creates cpusets only for measurements and housekeeping, not for loads. Loads use taskset for CPU affinity instead.
Changes:
- rteval-cmd: Update --cpusets help text to accurately describe that only measurement and housekeeping workloads get cpusets
- rteval/cpusetmanager.py: Update module docstring to clarify that loads use taskset rather than cpusets
This aligns the documentation with the simplified implementation approach that mirrors isolcpus philosophy: measurements get isolation, loads use affinity constraints.
--- rteval-cmd | 2 +- rteval/cpusetmanager.py | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
rteval-cmd +1 -1
@@ -118,7 +118,7 @@ def parse_options(cfg, parser, cmdargs): # CPU Configuration Options parser.add_argument("--cpusets", dest="rteval___cpusets", action="store_true", default=False, - help="use cgroup v2 cpusets to isolate measurement, load, and housekeeping workloads") + help="use cgroup v2 cpusets to isolate measurement and housekeeping workloads") parser.add_argument("--housekeeping", dest="rteval___housekeeping", type=str, default="", metavar="CPULIST", help="isolated CPUs reserved for system tasks (not used by rteval)")
rteval/cpusetmanager.py +2 -1
@@ -5,7 +5,8 @@ Manager for rteval cpusets with automatic cleanup This module provides the CpusetManager class which orchestrates cpuset creation -and process migration for rteval workloads (housekeeping, measurement, and loads). +and process migration for rteval workloads (housekeeping and measurement). +Loads use taskset for CPU affinity rather than cpusets. """ import time
Signature · 1 line
2.54.0
PATCH 7/7 rteval: Warn when all CPUs isolated and loads have no CPUs
In reply to John KacurAdd a warning when all online CPUs are isolated (via isolcpus boot parameter) and the loads cpulist is empty. This edge case occurs when a user isolates all CPUs but doesn't explicitly specify where loads should run, resulting in loads having no CPUs available.
The warning guides users to specify --loads-cpulist explicitly in this rare configuration.
Example scenario that triggers the warning:
- Boot parameter: isolcpus=0-15 (all CPUs)
- Command: rteval --cpusets -d 60s (no explicit CPU lists)
- Result: Loads default cpulist would be empty
The warning appears as:
"All CPUs are isolated and loads have no CPUs assigned.
Specify --loads-cpulist explicitly to choose where loads should run."
--- rteval-cmd | 9 +++++++++ 1 file changed, 9 insertions(+)
rteval-cmd +9 -0
@@ -477,6 +477,15 @@ if __name__ == '__main__': tmplist = [cpu for cpu in tmplist if cpu not in housekeeping_cpus] msrcfg.cpulist = collapse_cpulist(tmplist) + # Warn if all CPUs are isolated and loads have no CPUs + if not ldcfg.cpulist: + isolated_cpus = SysTopology().isolated_cpus() + online_cpus = SysTopology().online_cpus() + if isolated_cpus and set(isolated_cpus) == set(online_cpus): + logger.log(Log.WARN, + "All CPUs are isolated and loads have no CPUs assigned. " + "Specify --loads-cpulist explicitly to choose where loads should run.") + if ldcfg_cpulist_present: logger.log(Log.DEBUG, f"loads cpulist: {ldcfg.cpulist}") # if --onlyload is specified msrcfg.cpulist is unused
Signature · 1 line
2.54.0