rt-tests: hwlatdetect: Add MTBF calculation

Hwlatdetect reports the number of latency spikes but provides no information about their frequency distribution over time. This makes it difficult to compare results - a test with 10 spikes over 1 hour is very different from 10 spikes over 24 hours, but both show 'spikes = 10'.

Add Mean Time Between Failures (MTBF) calculation to quantify spike frequency.

By definition MTBF = total operating time / number of failures.

When the failure interval is large relative to test duration, this formula is biased. For example, imagine stable periodic failures. The total operating time will include time before the first failure and after the last failure. These intervals are determined by when the test starts and stops, not by the system's failure behavior, which adds measurement bias. The resulting MTBF will vary between runs even for stable periodic failures.

To reduce this bias, calculate MTBF using only the time between the first and last failure, divided by the number of intervals (failures minus one):

MTBF = (timestamp of last failure - timestamp of first failure) / (number of failures - 1)

In hwlatdetect, the failures are called samples.

This metric enables meaningful comparison of real-time performance consistency across different test runs, hardware configurations, and kernel versions. It can be considered a KPI for real-time stability, relevant for certification and SLA evaluation.

---

v3:
- Fix formatting
- Make first and last instance variables
v2:
- Use another more stable calculation of MTBF

---
 src/hwlatdetect/hwlatdetect.py | 7 +++++++
 1 file changed, 7 insertions(+)

index 38671724f3e3..bb12019ef2d6 100755
@@ -306,6 +306,8 @@     def __init__(self):
             raise DetectorNotAvailable("hwlat", "hwlat tracer not available")
         self.type = "tracer"
         self.samples = []
+        self.first = None
+        self.last = None
         self.set("enable", 0)
         self.set('current_tracer', 'hwlat')
 

@@ -338,6 +340,8 @@     def detect(self):
                 pollcnt += 1
                 val = self.get_sample()
                 while val:
+                    self.first = self.first or val.timestamp
+                    self.last = val.timestamp
                     self.samples.append(val)
                     if watch:
                         val.display()

@@ -559,6 +563,9 @@     def cleanup(self):
     exceeding = detect.get("count")
     info(f"Samples exceeding threshold: {exceeding}")
 
+    if exceeding > 1:
+        info(f"MTBF: {(float(detect.last) - float(detect.first)) / (exceeding - 1):.3f} seconds")
+
     if detect.have_msr:
         finishsmi = detect.getsmicounts()
         total_smis = 0

2.53.0

On Fri, 20 Feb 2026, Costa Shulyupin wrote:

8< -----------

    info(f"MTBF: {(float(detect.last) - float(detect.first)) / (len(detect.samples) - 1):.3f} seconds")

John

I disagree. Naturally, a longer sampling width increases the error. While using the total number of samples instead of the number of failures might appear more deterministic (since it avoids the resolution error inherent in width length), it fails to estimate MTBF. Such a metric becomes proportional to the duration of the sampling width (by several failures per sample width), whereas a true MTBF should remain independent of it.

Your question has led me to consider how non-sampling periods contribute to the error margin. The adjusted MTBF should therefore be compensated by multiplying it by the ratio of window to width. I’ll send an updated patch that includes this compensation factor.

Thanks
Costa