Skip to contentSkip to conversation

rteval: overflow bucket support and misc fixes

PATCH New John Kacur 1 participant 8 messages Jul 23

lore
Share & tools
lore

John Kacur

author
#0 lore raw

cover rteval: overflow bucket support and misc fixes

This series adds overflow bucket handling to cyclictest and timerlat measurement modules to properly track latencies that exceed the histogram range. Previously, these samples were silently dropped, making it difficult to assess system behavior under extreme latency conditions.

The overflow bucket is reported in both XML output and summary statistics, with warnings when latencies exceed the configured histogram range.

Also included are a license update for test files (GPL-2.0-only to GPL-2.0-or-later) and a fix for stressng --taskset argument parsing.

John Kacur (6):
rteval: Change test file licenses from GPL-2.0-only to

    GPL-2.0-or-later

rteval: Fix stressng --taskset argument parsing
cyclictest: Add overflow bucket to handle latencies exceeding
histogram range
rteval: Improve overflow bucket handling and add timerlat support
rteval: Refactor timerlat parsing and fix max line duplication bug
rteval: Reduce verbose debug logging in timerlat stoptrace parsing

 rteval/modules/loads/stressng.py         |   2 +-
 rteval/modules/measurement/cyclictest.py |  75 +++++++++++--
 rteval/modules/measurement/timerlat.py   | 131 ++++++++++++++++++++---
 rteval/rteval_text.xsl                   |  30 +++++-
 tests/README.md                          |   2 +-
 tests/test_cpusetmanager.py              |   2 +-
 tests/test_process_blocklist.py          |   2 +-
 tests/unittest-legacy.py                 |   2 +-
 8 files changed, 221 insertions(+), 25 deletions(-)
Signature · 1 line
2.55.0
author
#1 lore raw

PATCH 1/6 rteval: Change test file licenses from GPL-2.0-only to GPL-2.0-or-later

All test files were incorrectly marked as GPL-2.0-only when they should be GPL-2.0-or-later to match the rest of the codebase. This makes the licensing consistent across all rteval source files.

Updated files:
- tests/README.md
- tests/test_cpusetmanager.py
- tests/test_process_blocklist.py
- tests/unittest-legacy.py

---
 tests/README.md                 | 2 +-
 tests/test_cpusetmanager.py     | 2 +-
 tests/test_process_blocklist.py | 2 +-
 tests/unittest-legacy.py        | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)
tests/README.md +1 -1
index cac4f0e4021f..1a79cd3f578b 100644
@@ -166,7 +166,7 @@ Create a new test file following this pattern:
 
 ```python
 #!/usr/bin/env python3
-# SPDX-License-Identifier: GPL-2.0-only
+# SPDX-License-Identifier: GPL-2.0-or-later
 # Copyright 2026 John Kacur <[email protected]>
 """
 Description of what this test module tests.
tests/test_cpusetmanager.py +1 -1
index 6501b9a022d4..984e0f900edf 100644
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# SPDX-License-Identifier: GPL-2.0-only
+# SPDX-License-Identifier: GPL-2.0-or-later
 # Copyright 2026 John Kacur <[email protected]>
 """
 Unit tests for rteval CpusetManager functionality.
tests/test_process_blocklist.py +1 -1
index a568ea74478f..3c528c9607df 100644
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# SPDX-License-Identifier: GPL-2.0-only
+# SPDX-License-Identifier: GPL-2.0-or-later
 # Copyright 2026 John Kacur <[email protected]>
 """
 Unit tests for process blocklist functionality in rteval.
tests/unittest-legacy.py +1 -1
index 00811d7454ef..9e5ba3d0e831 100644
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# SPDX-License-Identifier: GPL-2.0-only
+# SPDX-License-Identifier: GPL-2.0-or-later
 # Copyright (c) 2010 Red Hat, Inc. All rights reserved.
 #
 # Author: David Sommerseth <[email protected]>
Signature · 1 line
2.55.0
author
#2 lore raw

PATCH 2/6 rteval: Fix stressng --taskset argument parsing

In reply to John Kacur

The --taskset option and its CPU list argument were being combined into a single string argument ('--taskset 0,1,2'), but stress-ng expects them as separate arguments ('--taskset', '0,1,2').

This caused stress-ng to fail to parse the --taskset option when a cpulist was specified.

Fix by using extend() instead of append() to add both the option and its value as separate list elements.

Assisted-by Claude:claude-sonnet-4-5
---
 rteval/modules/loads/stressng.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
rteval/modules/loads/stressng.py +1 -1
index 4ad7197fc590..387b5718a08f 100644
@@ -82,7 +82,7 @@ class Stressng(CommandLineLoad):
         if self.cpulist:
             for node in nodes:
                 cpulist = ",".join([str(n) for n in cpus[node]])
-                self.args.append(f'--taskset {cpulist}')
+                self.args.extend(['--taskset', cpulist])
 
     def _WorkloadTask(self):
         """ Kick of the workload here """
Signature · 1 line
2.55.0
author
#3 lore raw

PATCH 1/1 rteval: Fix measurement module cpuset migration and debug logging

In reply to John Kacur

This commit fixes two issues:

1. Cpuset migration disrupting measurement module initialization

Problem:
When using --cpusets, measurement modules (timerlat/cyclictest) were launched in the root cgroup, then migrated to the measurement cpuset after a 0.5s delay. This mid-startup migration disrupted per-CPU thread initialization, causing some CPUs to fail initialization. Symptoms included missing CPUs in output and zero samples collected.

Solution:
- Pass the measurement cpuset path to measurement modules before Start()
- Use subprocess preexec_fn to move child process into cpuset before exec
- Both timerlat and cyclictest now launch inside the cpuset from the start
- Fix is defensive: only activates when cpusets are configured
- Works correctly both with and without --cpusets flag

Files changed:
- rteval/__init__.py: Pass cpuset path to measurement modules
- rteval/modules/measurement/timerlat.py: Launch inside cpuset via preexec_fn
- rteval/modules/measurement/cyclictest.py: Launch inside cpuset via preexec_fn

2. Excessive debug logging from hackbench

Problem:
hackbench logs at DEBUG level every time it restarts (continuous loop), flooding debug output with thousands of identical messages.

Solution:
- Add 'initial' parameter to __starton() method
- Only log at DEBUG level for first startup
- Subsequent restarts don't spam debug log

Files changed:
- rteval/modules/loads/hackbench.py: Add initial parameter to reduce logging

Tested with:
- timerlat with cpusets (5 CPUs, 45M+ samples)
- cyclictest with cpusets (5 CPUs, 15M+ samples)
- timerlat without cpusets (16 CPUs, 142M+ samples)

Assisted-by Claude:claude-sonnet-4-5
---
 rteval/__init__.py                       | 14 ++++++++++++++
 rteval/modules/loads/hackbench.py        | 10 ++++++----
 rteval/modules/measurement/cyclictest.py | 17 ++++++++++++++++-
 rteval/modules/measurement/timerlat.py   | 17 ++++++++++++++++-
 4 files changed, 52 insertions(+), 6 deletions(-)
rteval/__init__.py +14 -0
index 6e49889a6d67..b6d3c39a4b6a 100644
@@ -174,6 +174,20 @@ class RtEval(rtevalReport):
                 print(f"started measurement threads on {onlinecpus} cores")
             print(f"Run duration: {str(self.__rtevcfg.duration)} seconds")
 
+            # Pass cpuset path to measurement modules so they can launch inside the cpuset
+            if self._cpuset_manager and self._cpuset_manager.measurement_cpuset:
+                cpuset_path = self._cpuset_manager.measurement_cpuset._cpuset_path
+                for (modname, mod) in self._measuremods._RtEvalModules__modules:
+                    # Try to find the config attribute - modules use self.__cfg (name-mangled)
+                    cfg = None
+                    for attr in ['_cfg', f'_{mod.__class__.__name__}__cfg']:
+                        if hasattr(mod, attr):
+                            cfg = getattr(mod, attr)
+                            break
+                    if cfg:
+                        cfg.cpuset_path = cpuset_path
+                        self.__logger.log(Log.DEBUG, f"Set cpuset_path for {modname}: {cpuset_path}")
+
             self._measuremods.Start()
 
             # Unleash the loads and measurement threads
rteval/modules/loads/hackbench.py +6 -4
index 946a2adc2e57..20ebbbc4617c 100644
@@ -110,7 +110,7 @@ class Hackbench(CommandLineLoad):
 
         self.started = False
 
-    def __starton(self, node):
+    def __starton(self, node, initial=False):
         if self.__multinodes or self.cpulist:
             if self.__usenumactl:
                 args = ['numactl', '--cpunodebind', str(node)] + self.args
@@ -120,7 +120,9 @@ class Hackbench(CommandLineLoad):
         else:
             args = self.args
 
-        self._log(Log.DEBUG, f"starting on node {node}: args = {args}")
+        # Only log initial startup at DEBUG level to avoid spam
+        if initial:
+            self._log(Log.DEBUG, f"starting on node {node}: args = {args}")
         p = subprocess.Popen(args,
                              stdin=self.__nullfp,
                              stdout=self.__out,
@@ -137,7 +139,7 @@ class Hackbench(CommandLineLoad):
         # just do this once
         if not self.started:
             for n in self.nodes:
-                self.tasks[n] = self.__starton(n)
+                self.tasks[n] = self.__starton(n, initial=True)
             self.started = True
             return
 
@@ -145,7 +147,7 @@ class Hackbench(CommandLineLoad):
             try:
                 if self.tasks[n].poll() is not None:
                     self.tasks[n].wait()
-                    self.tasks[n] = self.__starton(n)
+                    self.tasks[n] = self.__starton(n, initial=False)
             except OSError as e:
                 if e.errno != errno.ENOMEM:
                     raise e
rteval/modules/measurement/cyclictest.py +16 -1
index a1bf0d2d7324..eee315d270f8 100644
@@ -302,10 +302,25 @@ class Cyclictest(rtevalModulePrototype):
         if not self._logging:
             self.__cyclicoutput.seek(0)
 
+        # If cpuset is configured, launch process inside the cpuset
+        # This is critical for cyclictest - migrating it mid-startup disrupts initialization
+        preexec_fn = None
+        if hasattr(self.__cfg, 'cpuset_path') and self.__cfg.cpuset_path:
+            def move_to_cpuset():
+                """Move child process into cpuset before exec"""
+                try:
+                    cpuset_procs = os.path.join(self.__cfg.cpuset_path, 'cgroup.procs')
+                    with open(cpuset_procs, 'w') as f:
+                        f.write(str(os.getpid()))
+                except Exception:
+                    pass  # Fail silently - parent will attempt migration as fallback
+            preexec_fn = move_to_cpuset
+
         self.__cyclicprocess = subprocess.Popen(self.__cmd,
                                                 stdout=self.__cyclicoutput,
                                                 stderr=self.__nullfp,
-                                                stdin=self.__nullfp)
+                                                stdin=self.__nullfp,
+                                                preexec_fn=preexec_fn)
         self.__started = True
 
     def WorkloadAlive(self):
rteval/modules/measurement/timerlat.py +16 -1
index 7f592ea697af..418f1ba151af 100644
@@ -285,10 +285,25 @@ class Timerlat(rtevalModulePrototype):
             self.__timerlat_out.seek(0)
             self.__timerlat_err.seek(0)
 
+        # If cpuset is configured, launch process inside the cpuset
+        # This is critical for timerlat - migrating it mid-startup disrupts initialization
+        preexec_fn = None
+        if hasattr(self.__cfg, 'cpuset_path') and self.__cfg.cpuset_path:
+            def move_to_cpuset():
+                """Move child process into cpuset before exec"""
+                try:
+                    cpuset_procs = os.path.join(self.__cfg.cpuset_path, 'cgroup.procs')
+                    with open(cpuset_procs, 'w') as f:
+                        f.write(str(os.getpid()))
+                except Exception:
+                    pass  # Fail silently - parent will attempt migration as fallback
+            preexec_fn = move_to_cpuset
+
         self.__timerlat_process = subprocess.Popen(self.__cmd,
                                                    stdout=self.__timerlat_out,
                                                    stderr=self.__timerlat_err,
-                                                   stdin=None)
+                                                   stdin=None,
+                                                   preexec_fn=preexec_fn)
         self.__started = True
 
     def WorkloadAlive(self):
Signature · 1 line
2.55.0
author
#4 lore raw

PATCH 3/6 cyclictest: Add overflow bucket to handle latencies exceeding histogram range

In reply to John Kacur

Previously, when maximum latency exceeded the configured histogram range (buckets parameter), rteval would skip generating statistics entirely and return an error. This was problematic, for example when running on non-RT systems where occasional latency spikes are expected.

This commit implements an overflow bucket mechanism:

- Samples exceeding the histogram range are clamped to the last bucket
- The actual maximum latency is still tracked and reported
- Overflow sample count is tracked and reported in XML output
- Statistics are now generated even when max > buckets
- Error messages changed to warnings with overflow information

Benefits:
- Users get statistics even with latency spikes on non-RT systems
- No need to guess the right bucket size in advance
- Clear indication when overflow occurs via warnings and XML metadata
- XML output includes overflow_samples count and overflow_bucket index

Example XML output when overflow occurs:
<statistics>
...
<overflow_samples info="42 samples exceeded histogram range">42</overflow_samples>
</statistics>
<histogram nbuckets="3500" overflow_bucket="3499" overflow_count="42">

    ...

</histogram>

This addresses the issue where rteval would fail to produce useful data, for example when running on non-RT kernels with occasional high latency spikes.

---
 rteval/modules/measurement/cyclictest.py | 38 +++++++++++++++++++++---
 1 file changed, 34 insertions(+), 4 deletions(-)
rteval/modules/measurement/cyclictest.py +34 -4
index eee315d270f8..8371aa2e2aa5 100644
@@ -38,6 +38,8 @@ class RunData:
         self.__range = 0.0
         self.__mad = 0.0
         self._log = logfnc
+        self.__overflow_samples = 0  # Track samples that exceeded bucket range
+        self.__bucket_limit = None   # Maximum bucket index
 
     def __str__(self):
         retval = f"id:         {self.__id}\n"
@@ -61,11 +63,28 @@ class RunData:
         if value < self.__min:
             self.__min = value
 
+    def set_bucket_limit(self, limit):
+        """Set the maximum bucket index for overflow handling"""
+        self.__bucket_limit = limit
+
+    def get_overflow_count(self):
+        """Get the number of samples that exceeded the bucket limit"""
+        return self.__overflow_samples
+
     def bucket(self, index, value):
-        self.__samples[index] = self.__samples.setdefault(index, 0) + value
+        """Add samples to histogram bucket, clamping overflow to the last bucket"""
+        # Track the actual max before clamping
         if value:
             self.update_max(index)
             self.update_min(index)
+
+        # Clamp to bucket limit if set (overflow bucket)
+        if self.__bucket_limit is not None and index >= self.__bucket_limit:
+            if value:
+                self.__overflow_samples += value
+            index = self.__bucket_limit - 1  # Use last bucket as overflow bucket
+
+        self.__samples[index] = self.__samples.setdefault(index, 0) + value
         self.__numsamples += value
 
     def reduce(self):
@@ -166,8 +185,16 @@ class RunData:
             n = stat_n.newTextChild(None, 'standard_deviation', str(self.__stddev))
             n.newProp('unit', 'us')
 
+            # Report overflow samples if any
+            if self.__overflow_samples > 0:
+                n = stat_n.newTextChild(None, 'overflow_samples', str(self.__overflow_samples))
+                n.newProp('info', f'{self.__overflow_samples} samples exceeded histogram range')
+
             hist_n = rep_n.newChild(None, 'histogram', None)
             hist_n.newProp('nbuckets', str(len(self.__samples)))
+            if self.__overflow_samples > 0:
+                hist_n.newProp('overflow_bucket', str(self.__bucket_limit - 1))
+                hist_n.newProp('overflow_count', str(self.__overflow_samples))
             keys = list(self.__samples.keys())
             keys.sort()
             for k in keys:
@@ -205,12 +232,14 @@ class Cyclictest(rtevalModulePrototype):
             self.__cyclicdata[core] = RunData(core, 'core', self.__priority,
                                               logfnc=self._log)
             self.__cyclicdata[core].description = info[core]['model name']
+            self.__cyclicdata[core].set_bucket_limit(self.__buckets)
 
         # Create a RunData object for the overall system
         self.__cyclicdata['system'] = RunData('system',
                                               'system', self.__priority,
                                               logfnc=self._log)
         self.__cyclicdata['system'].description = (f"({self.__numcores} cores) ") + info['0']['model name']
+        self.__cyclicdata['system'].set_bucket_limit(self.__buckets)
 
         # Logging configuration
         self._logging = self.__cfg.setdefault('logging', False)
@@ -480,9 +509,10 @@ class Cyclictest(rtevalModulePrototype):
 
         # Let the user know if max latency overshot the number of buckets
         if self.__cyclicdata["system"].get_max() > self.__buckets:
-            self._log(Log.ERR, f'Max latency({self.__cyclicdata["system"].get_max()}us) exceeded histogram range({self.__buckets}us). Skipping statistics')
-            self._log(Log.ERR, "Increase number of buckets to avoid lost samples")
-            return rep_n
+            overflow_count = self.__cyclicdata["system"].get_overflow_count()
+            self._log(Log.WARN, f'Max latency({self.__cyclicdata["system"].get_max()}us) exceeded histogram range({self.__buckets}us)')
+            self._log(Log.WARN, f'{overflow_count} samples clamped to overflow bucket at index {self.__buckets - 1}')
+            self._log(Log.WARN, "Consider increasing buckets parameter for better resolution")
 
         rep_n.addChild(self.__cyclicdata["system"].MakeReport())
         for thr in self.__cpus:
Signature · 1 line
2.55.0
author
#5 lore raw

PATCH 4/6 rteval: Improve overflow bucket handling and add timerlat support

In reply to John Kacur

This improves the overflow bucket implementation by using cyclictest's native overflow tracking instead of manual bucket clamping, and extends overflow support to the timerlat measurement module.

Changes to cyclictest module:
- Parse "# Histogram Overflows:" line from cyclictest output
- Remove manual bucket clamping logic (cyclictest handles this)
- Improve SIGINT timeout handling with progressive waits (10s->5s->2s)
- Create dedicated overflow bucket with index="overflow" attribute
- Remove overflow_bucket attribute (no longer needed)

Changes to timerlat module:
- Parse "over:" line from rtla timerlat hist output
- Track overflow per-core and accumulate for system-wide totals
- Add overflow_samples to statistics output
- Improve SIGINT timeout handling matching cyclictest
- Create dedicated overflow bucket with index="overflow" attribute
- Log warnings when overflow occurs

Changes to XSL stylesheet:
- Display total overflow count in summary section
- Display per-core overflow samples in statistics output

This provides consistent overflow handling across both measurement modules, giving users meaningful statistics even when latency spikes exceed the histogram range.

Assisted-by Claude:claude-sonnet-4-5
---
 rteval/modules/measurement/cyclictest.py | 75 +++++++++++++++-------
 rteval/modules/measurement/timerlat.py   | 79 +++++++++++++++++++++++-
 rteval/rteval_text.xsl                   | 30 ++++++++-
 3 files changed, 159 insertions(+), 25 deletions(-)
rteval/modules/measurement/cyclictest.py +54 -21
index 8371aa2e2aa5..b193330e27e0 100644
@@ -39,7 +39,6 @@ class RunData:
         self.__mad = 0.0
         self._log = logfnc
         self.__overflow_samples = 0  # Track samples that exceeded bucket range
-        self.__bucket_limit = None   # Maximum bucket index
 
     def __str__(self):
         retval = f"id:         {self.__id}\n"
@@ -63,28 +62,24 @@ class RunData:
         if value < self.__min:
             self.__min = value
 
-    def set_bucket_limit(self, limit):
-        """Set the maximum bucket index for overflow handling"""
-        self.__bucket_limit = limit
+    def set_overflow_count(self, count):
+        """Set overflow count from cyclictest output"""
+        self.__overflow_samples = count
+
+    def add_overflow_samples(self, count):
+        """Add to overflow count (for system-wide accumulation)"""
+        self.__overflow_samples += count
 
     def get_overflow_count(self):
         """Get the number of samples that exceeded the bucket limit"""
         return self.__overflow_samples
 
     def bucket(self, index, value):
-        """Add samples to histogram bucket, clamping overflow to the last bucket"""
-        # Track the actual max before clamping
+        """Add samples to histogram bucket"""
+        self.__samples[index] = self.__samples.setdefault(index, 0) + value
         if value:
             self.update_max(index)
             self.update_min(index)
-
-        # Clamp to bucket limit if set (overflow bucket)
-        if self.__bucket_limit is not None and index >= self.__bucket_limit:
-            if value:
-                self.__overflow_samples += value
-            index = self.__bucket_limit - 1  # Use last bucket as overflow bucket
-
-        self.__samples[index] = self.__samples.setdefault(index, 0) + value
         self.__numsamples += value
 
     def reduce(self):
@@ -193,7 +188,6 @@ class RunData:
             hist_n = rep_n.newChild(None, 'histogram', None)
             hist_n.newProp('nbuckets', str(len(self.__samples)))
             if self.__overflow_samples > 0:
-                hist_n.newProp('overflow_bucket', str(self.__bucket_limit - 1))
                 hist_n.newProp('overflow_count', str(self.__overflow_samples))
             keys = list(self.__samples.keys())
             keys.sort()
@@ -205,6 +199,13 @@ class RunData:
                 b_n.newProp('index', str(k))
                 b_n.newProp('value', str(self.__samples[k]))
 
+            # Add dedicated overflow bucket if overflow occurred
+            if self.__overflow_samples > 0:
+                b_n = hist_n.newChild(None, 'bucket', None)
+                b_n.newProp('index', 'overflow')
+                b_n.newProp('value', str(self.__overflow_samples))
+                b_n.newProp('type', 'overflow')
+
         return rep_n
 
 
@@ -232,14 +233,12 @@ class Cyclictest(rtevalModulePrototype):
             self.__cyclicdata[core] = RunData(core, 'core', self.__priority,
                                               logfnc=self._log)
             self.__cyclicdata[core].description = info[core]['model name']
-            self.__cyclicdata[core].set_bucket_limit(self.__buckets)
 
         # Create a RunData object for the overall system
         self.__cyclicdata['system'] = RunData('system',
                                               'system', self.__priority,
                                               logfnc=self._log)
         self.__cyclicdata['system'].description = (f"({self.__numcores} cores) ") + info['0']['model name']
-        self.__cyclicdata['system'].set_bucket_limit(self.__buckets)
 
         # Logging configuration
         self._logging = self.__cfg.setdefault('logging', False)
@@ -377,19 +376,51 @@ class Cyclictest(rtevalModulePrototype):
         except (IndexError, ValueError) as e:
             self._log(Log.WARN, f"Error parsing max latencies: {e}")
 
+    def _parse_histogram_overflows(self, line):
+        if not line.startswith('# Histogram Overflows:'):
+            return
+
+        try:
+            line = line.split(':')[1]
+            vals = [int(x) for x in line.split()]
+
+            # First N values are per-core overflow counts
+            for i, core in enumerate(self.__cpus):
+                if i < len(vals):
+                    overflow_count = vals[i]
+                    self.__cyclicdata[core].set_overflow_count(overflow_count)
+                    self.__cyclicdata['system'].add_overflow_samples(overflow_count)
+        except (IndexError, ValueError) as e:
+            self._log(Log.WARN, f"Error parsing histogram overflows: {e}")
 
     def _WorkloadCleanup(self):
         if not self.__started:
             return
 
         # Send SIGINT and wait for graceful exit
-        # Limit attempts to avoid infinite loop (RHEL-140898)
+        # Use wait(timeout) to return immediately when process exits
         max_attempts = 5
         attempt = 0
         while self.__cyclicprocess.poll() is None and attempt < max_attempts:
             self._log(Log.DEBUG, f"Sending SIGINT (attempt {attempt + 1}/{max_attempts})")
             os.kill(self.__cyclicprocess.pid, signal.SIGINT)
-            time.sleep(2)
+
+            # Use longer timeout on first attempt to allow histogram output to complete
+            # Subsequent attempts use shorter timeouts
+            if attempt == 0:
+                timeout = 10  # First attempt: 10 seconds for histogram output
+            elif attempt == 1:
+                timeout = 5   # Second attempt: 5 seconds
+            else:
+                timeout = 2   # Remaining attempts: 2 seconds
+
+            try:
+                self.__cyclicprocess.wait(timeout=timeout)
+                break  # Process exited gracefully
+            except subprocess.TimeoutExpired:
+                # Process still running after timeout, will try again
+                pass
+
             attempt += 1
 
         # Check if process exited
@@ -431,6 +462,8 @@ class Cyclictest(rtevalModulePrototype):
                             self._log(Log.WARN, f"Error parsing break value: {e}")
                     elif line.startswith('# Max Latencies: '):
                         self._parse_max_latencies(line)
+                    elif line.startswith('# Histogram Overflows:'):
+                        self._parse_histogram_overflows(line)
                     continue
 
                 # Skipping blank lines
@@ -508,10 +541,10 @@ class Cyclictest(rtevalModulePrototype):
             rep_n.addChild(abrt_n)
 
         # Let the user know if max latency overshot the number of buckets
-        if self.__cyclicdata["system"].get_max() > self.__buckets:
+        if self.__cyclicdata["system"].get_max() >= self.__buckets:
             overflow_count = self.__cyclicdata["system"].get_overflow_count()
             self._log(Log.WARN, f'Max latency({self.__cyclicdata["system"].get_max()}us) exceeded histogram range({self.__buckets}us)')
-            self._log(Log.WARN, f'{overflow_count} samples clamped to overflow bucket at index {self.__buckets - 1}')
+            self._log(Log.WARN, f'{overflow_count} samples stored in overflow bucket')
             self._log(Log.WARN, "Consider increasing buckets parameter for better resolution")
 
         rep_n.addChild(self.__cyclicdata["system"].MakeReport())
rteval/modules/measurement/timerlat.py +76 -3

128 lines not on this page · raw message

rteval/rteval_text.xsl +29 -1
index da7084289f87..a6829119170b 100644
@@ -163,7 +163,23 @@
           <xsl:text>not present</xsl:text>
         </xsl:otherwise>
     </xsl:choose>
-    <xsl:text>&#10;&#10;</xsl:text>
+    <xsl:text>&#10;</xsl:text>
+
+    <!-- Display total overflow samples if present -->
+    <xsl:if test="Measurements/cyclictest/system/statistics/overflow_samples or Measurements/timerlat/system/statistics/overflow_samples">
+      <xsl:text>   Total Overflows: </xsl:text>
+      <xsl:choose>
+        <xsl:when test="Measurements/cyclictest/system/statistics/overflow_samples">
+          <xsl:value-of select="Measurements/cyclictest/system/statistics/overflow_samples"/>
+        </xsl:when>
+        <xsl:when test="Measurements/timerlat/system/statistics/overflow_samples">
+          <xsl:value-of select="Measurements/timerlat/system/statistics/overflow_samples"/>
+        </xsl:when>
+      </xsl:choose>
+      <xsl:text> samples exceeded histogram range</xsl:text>
+      <xsl:text>&#10;</xsl:text>
+    </xsl:if>
+    <xsl:text>&#10;</xsl:text>
 
     <!-- Display core sharing warnings if present -->
     <xsl:if test="SystemInfo/CPUtopology/CoreSharingWarnings/warning">
@@ -321,6 +337,12 @@
       <xsl:value-of select="standard_deviation"/>
       <xsl:value-of select="standard_deviation/@unit"/>
       <xsl:text>&#10;</xsl:text>
+
+      <xsl:if test="overflow_samples">
+        <xsl:text>            Overflow samples:  </xsl:text>
+        <xsl:value-of select="overflow_samples"/>
+        <xsl:text>&#10;</xsl:text>
+      </xsl:if>
     </xsl:if>
     <xsl:text>&#10;</xsl:text>
   </xsl:template>
@@ -428,6 +450,12 @@
       <xsl:value-of select="standard_deviation"/>
       <xsl:value-of select="standard_deviation/@unit"/>
       <xsl:text>&#10;</xsl:text>
+
+      <xsl:if test="overflow_samples">
+        <xsl:text>            Overflow samples:  </xsl:text>
+        <xsl:value-of select="overflow_samples"/>
+        <xsl:text>&#10;</xsl:text>
+      </xsl:if>
     </xsl:if>
     <xsl:text>&#10;</xsl:text>
   </xsl:template>
Signature · 1 line
2.55.0
author
#6 lore raw

PATCH 5/6 rteval: Refactor timerlat parsing and fix max line duplication bug

In reply to John Kacur

Extract max and overflow parsing into dedicated helper methods (_parse_max_latencies and _parse_histogram_overflows) for better code organization and consistency with cyclictest.py structure.

Fix bug where rtla timerlat's ALL: section contains a second max: line with only 3 aggregate values (IRQ, Thread, User maxima across all cores). This second max: line was incorrectly being parsed as core 0 data, overwriting the correct per-core maximum values from the first max: line.

The first max: line contains per-core data (48 values for 16 cores with 3 columns each), while the second is an aggregate summary. Add max_parsed flag to track parsing state and only parse the first occurrence.

When stoptrace is disabled, break after parsing the first max: line since all needed histogram data has been collected. When stoptrace is enabled, continue parsing to capture trace analysis data but skip the duplicate
max: line.

Enable rtla timerlat summary output (remove --no-summary) to get accurate max values from the summary line rather than calculating from histogram buckets, which are capped at the bucket range.

Assisted-by Claude:claude-sonnet-4-5
---
 rteval/modules/measurement/timerlat.py | 90 +++++++++++++++++---------
 1 file changed, 61 insertions(+), 29 deletions(-)
rteval/modules/measurement/timerlat.py +61 -29
index 1201302d47b4..f983cedf0daa 100644
@@ -256,6 +256,54 @@ class Timerlat(rtevalModulePrototype):
             os.makedirs(logdir)
         return os.open(os.path.join(logdir, name), os.O_CREAT|os.O_RDWR|os.O_TRUNC)
 
+    def _parse_max_latencies(self, line):
+        """Parse the max: line from rtla timerlat output"""
+        if not line.startswith('max:'):
+            return
+
+        try:
+            vals = line.split()
+            if not vals or vals[0] != 'max:':
+                return
+
+            for i, core in enumerate(self.__cpus):
+                # timerlat has 3 columns per core: IRQ, Thread, User
+                # Take the maximum of the 3 values as the overall max for this core
+                if i*3 + 3 < len(vals):
+                    irq_max = int(vals[i*3+1])
+                    thr_max = int(vals[i*3+2])
+                    usr_max = int(vals[i*3+3])
+                    core_max = max(irq_max, thr_max, usr_max)
+
+                    self.__timerlatdata[core].update_max(core_max)
+                    self.__timerlatdata['system'].update_max(core_max)
+        except (IndexError, ValueError) as e:
+            self._log(Log.DEBUG, f"Error parsing max line: {e}")
+
+    def _parse_histogram_overflows(self, line):
+        """Parse the over: line from rtla timerlat output"""
+        if not line.startswith('over:'):
+            return
+
+        try:
+            vals = line.split()
+            if not vals or vals[0] != 'over:':
+                return
+
+            for i, core in enumerate(self.__cpus):
+                # timerlat has 3 columns per core: IRQ, Thread, User
+                # Calculate total overflow for this core
+                if i*3 + 3 < len(vals):
+                    irq_overflow = int(vals[i*3+1])
+                    thr_overflow = int(vals[i*3+2])
+                    usr_overflow = int(vals[i*3+3])
+                    total_overflow = irq_overflow + thr_overflow + usr_overflow
+
+                    self.__timerlatdata[core].set_overflow_count(total_overflow)
+                    self.__timerlatdata['system'].add_overflow_samples(total_overflow)
+        except (IndexError, ValueError) as e:
+            self._log(Log.DEBUG, f"Error parsing overflow line: {e}")
+
     def _WorkloadSetup(self):
         self.__timerlat_process = None
 
@@ -271,7 +319,7 @@ class Timerlat(rtevalModulePrototype):
         self.__cmd.extend(['-P', f'f:{int(self.__priority)}', '-u'])
         self.__cmd.extend(['-c', self.__cpulist])
         self.__cmd.extend(['-E', str(self.__buckets)])
-        self.__cmd.append('--no-summary')
+        # self.__cmd.append('--no-summary')  # Commented out to get max: line for accurate overflow max values
         # Only disable auto-analysis if not using stoptrace (stoptrace needs it for trace analysis output)
         if not self.__cfg.stoptrace:
             self.__cmd.append('--no-aa')
@@ -405,6 +453,7 @@ class Timerlat(rtevalModulePrototype):
             blocking_thread_detected = False
             softirq_interference_detected = False
             irq_interference_detected = False
+            max_parsed = False  # Track if we've already parsed the max: line
 
             for line in output_lines:
                 if isinstance(line, bytes):
@@ -503,29 +552,7 @@ class Timerlat(rtevalModulePrototype):
                     #print(line)
                     continue
                 elif line.startswith('over:'):
-                    # Parse overflow counts: 3 values per CPU (IRQ, Thread, User)
-                    vals = line.split()
-                    if not vals or vals[0] != 'over:':
-                        continue
-                    try:
-                        for i, core in enumerate(self.__cpus):
-                            # timerlat has 3 columns per core: IRQ, Thread, User
-                            # Calculate total overflow for this core
-                            if i*3 + 3 < len(vals):
-                                irq_overflow = int(vals[i*3+1])
-                                thr_overflow = int(vals[i*3+2])
-                                usr_overflow = int(vals[i*3+3])
-                                total_overflow = irq_overflow + thr_overflow + usr_overflow
-
-                                self.__timerlatdata[core].set_overflow_count(total_overflow)
-                                self.__timerlatdata['system'].add_overflow_samples(total_overflow)
-
-                                # If overflow occurred, max latency is at least the bucket limit
-                                if total_overflow > 0:
-                                    self.__timerlatdata[core].update_max(self.__buckets)
-                                    self.__timerlatdata['system'].update_max(self.__buckets)
-                    except (IndexError, ValueError) as e:
-                        self._log(Log.DEBUG, f"Error parsing overflow line: {e}")
+                    self._parse_histogram_overflows(line)
                     continue
                 elif line.startswith('count:'):
                     #print(line)
@@ -537,16 +564,21 @@ class Timerlat(rtevalModulePrototype):
                     #print(line)
                     continue
                 elif line.startswith('max:'):
-                    #print(line)
+                    # Only parse the first max: line (per-core data)
+                    # Skip subsequent max: lines (like the ALL: section)
+                    if not max_parsed:
+                        self._parse_max_latencies(line)
+                        max_parsed = True
+                        # Max line is the last histogram data we need
+                        # If stoptrace is enabled, continue parsing for trace analysis
+                        # Otherwise stop to avoid the ALL: section
+                        if not self.__cfg.stoptrace:
+                            break
                     continue
                 elif line.startswith('timerlat hit stop tracing'):
                     self.__stoptrace = True
                     self.__posttrace += line
                     continue
-                elif line.startswith('ALL:'):
-                    # We should only see 'ALL:' without timerlat --no-summary
-                    # print(line)
-                    continue
                 else:
                     #print(line)
                     pass
Signature · 1 line
2.55.0
author
#7 lore raw

PATCH 6/6 rteval: Reduce verbose debug logging in timerlat stoptrace parsing

In reply to John Kacur

Comment out verbose debug logging that dumps entire data structures during stoptrace parsing. These were useful during initial development but create excessive output now that the feature is stable.

Commented logs can be easily re-enabled for debugging if needed:
- Full self.__stdata dictionary dump
- Per-item cpu/key/value logging during XML generation
- Complete posttrace output dump

Assisted-by Claude:claude-sonnet-4-5
---
 rteval/modules/measurement/timerlat.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
rteval/modules/measurement/timerlat.py +4 -4
index f983cedf0daa..735141e6796d 100644
@@ -653,7 +653,7 @@ class Timerlat(rtevalModulePrototype):
         rep_n.addChild(stoptrace_invoked_n)
 
         if self.stcpu != -1:
-            self._log(Log.DEBUG, f'self.__stdata = {self.__stdata}')
+            # self._log(Log.DEBUG, f'self.__stdata = {self.__stdata}')
             for cpu in self.__stdata:
                 # This is  Max timerlat IRQ latency from idle
                 # With no other data from that cpu, so don't create a
@@ -663,7 +663,7 @@ class Timerlat(rtevalModulePrototype):
                 stoptrace_n = libxml2.newNode('stoptrace_report')
                 stoptrace_n.newProp("CPU", str(cpu))
                 for k, v in self.__stdata[cpu].items():
-                    self._log(Log.DEBUG, f"cpu={cpu}, k={k}, v={v}")
+                    # self._log(Log.DEBUG, f"cpu={cpu}, k={k}, v={v}")
                     if isinstance(v, tuple):
                         latency = str(v[0])
                         percent = str(v[1])
@@ -687,8 +687,8 @@ class Timerlat(rtevalModulePrototype):
                         cpu_n.newProp('unit', 'us')
                 rep_n.addChild(stoptrace_n)
 
-            self._log(Log.DEBUG, f'timerlat: posttrace = \n{self.__posttrace}')
-            self._log(Log.DEBUG, 'timerlat: posttrace END')
+            # self._log(Log.DEBUG, f'timerlat: posttrace = \n{self.__posttrace}')
+            # self._log(Log.DEBUG, 'timerlat: posttrace END')
             for cpu in self.__stdata:
                 for k, v in self.__stdata[cpu].items():
                     if isinstance(v, tuple):
Signature · 1 line
2.55.0