Skip to contentSkip to conversation

rtla fixes for 6.5

PATCH New Daniel Bristot de Oliveira 1 participant 4 messages Aug 4, 2023 Real-time Linux Analysis (rtla) tools

lore
Share & tools
lore
#0 lore raw

cover rtla fixes for 6.5

Here are some fixes for rtla timerlat auto-analysis.

The first one zeroes the variable that accounts for the amount of thread interference when a new cycle begins.

The second fixes an imprecision on the IRQ delay account when the value is near 0.

The last one fixes the case when an IRQ interference for the next activation is considered for the previous one.

Daniel Bristot de Oliveira (3):
rtla/timerlat_aa: Zero thread sum after every sample analysis rtla/timerlat_aa: Fix negative irq delay rtla/timerlat_aa: Fix previous IRQ delay for IRQs that happens after
thread sample

 tools/tracing/rtla/src/timerlat_aa.c | 32 ++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)
Signature · 1 line
2.38.1
#1 lore raw

PATCH 1/3 rtla/timerlat_aa: Zero thread sum after every sample analysis

The thread thread_thread_sum accounts for thread interference during a single activation. It was not being zeroed, so it was accumulating thread interference over all activations.

It was not that visible when timerlat was the highest priority.

Fixes 27e348b221f6 ("rtla/timerlat: Add auto-analysis core")
---
 tools/tracing/rtla/src/timerlat_aa.c | 1 +
 1 file changed, 1 insertion(+)
tools/tracing/rtla/src/timerlat_aa.c +1 -0
index e0ffe69c271c..dec5b4c4511e 100644
@@ -159,6 +159,7 @@ static int timerlat_aa_irq_latency(struct timerlat_aa_data *taa_data,
 	taa_data->thread_nmi_sum = 0;
 	taa_data->thread_irq_sum = 0;
 	taa_data->thread_softirq_sum = 0;
+	taa_data->thread_thread_sum = 0;
 	taa_data->thread_blocking_duration = 0;
 	taa_data->timer_irq_start_time = 0;
 	taa_data->timer_irq_duration = 0;
Signature · 1 line
2.38.1
#2 lore raw

PATCH 2/3 rtla/timerlat_aa: Fix negative IRQ delay

In reply to Daniel Bristot de Oliveira

When estimating the IRQ timer delay, we are dealing with two different clock sources: the external clock source that timerlat uses as a reference and the clock used by the tracer. There are also two moments: the time reading the clock and the timer in which the event is placed in the buffer (the trace event timestamp).

If the processor is slow or there is some hardware noise, the difference between the timestamp and the external clock, read can be longer than the IRQ handler delay, resulting in a negative time.

If so, set IRQ to start delay as 0. In the end, it is less near-zero and relevant then the noise.

Fixes 27e348b221f6 ("rtla/timerlat: Add auto-analysis core")
---
 tools/tracing/rtla/src/timerlat_aa.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
tools/tracing/rtla/src/timerlat_aa.c +17 -1
index dec5b4c4511e..baf1efda0581 100644
@@ -338,7 +338,23 @@ static int timerlat_aa_irq_handler(struct trace_seq *s, struct tep_record *recor
 		taa_data->timer_irq_start_time = start;
 		taa_data->timer_irq_duration = duration;
 
-		taa_data->timer_irq_start_delay = taa_data->timer_irq_start_time - expected_start;
+		/*
+		 * We are dealing with two different clock sources: the
+		 * external clock source that timerlat uses as a reference
+		 * and the clock used by the tracer. There are also two
+		 * moments: the time reading the clock and the timer in
+		 * which the event is placed in the buffer (the trace
+		 * event timestamp). If the processor is slow or there
+		 * is some hardware noise, the difference between the
+		 * timestamp and the external clock read can be longer
+		 * than the IRQ handler delay, resulting in a negative
+		 * time. If so, set IRQ start delay as 0. In the end,
+		 * it is less relevant than the noise.
+		 */
+		if (expected_start < taa_data->timer_irq_start_time)
+			taa_data->timer_irq_start_delay = taa_data->timer_irq_start_time - expected_start;
+		else
+			taa_data->timer_irq_start_delay = 0;
 
 		/*
 		 * not exit from idle.
Signature · 1 line
2.38.1
#3 lore raw

PATCH 3/3 rtla/timerlat_aa: Fix previous IRQ delay for IRQs that happens after thread sample

In reply to Daniel Bristot de Oliveira

timerlat auto-analysis takes note of all IRQs, before or after the execution of the timerlat thread.

Because we cannot go backward in the trace (we will fix it when moving to trace-cmd lib?), timerlat aa take note of the last IRQ execution in the waiting for the IRQ state, and then print it if it is executed after the expected timer IRQ starting time.

After the thread sample, the timerlat starts recording the next IRQs as "previous" irq for the next occurrence.

However, if an IRQ happens after the thread measurement but before the tracing stops, it is classified as a previous IRQ. That is not wrong, as it can be "previous" for the subsequent activation. What is wrong is considering it as a potential source for the last activation.

Ignore the IRQ interference that happens after the IRQ starting time for now. A future improvement for timerlat can be either keeping a list of previous IRQ execution or using the trace-cmd library. Still, it requires further investigation - it is a new feature.

Fixes 27e348b221f6 ("rtla/timerlat: Add auto-analysis core")
---
 tools/tracing/rtla/src/timerlat_aa.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
tools/tracing/rtla/src/timerlat_aa.c +8 -5
index baf1efda0581..7093fd5333be 100644
@@ -545,7 +545,7 @@ static int timerlat_aa_kworker_start_handler(struct trace_seq *s, struct tep_rec
 static void timerlat_thread_analysis(struct timerlat_aa_data *taa_data, int cpu,
 				     int irq_thresh, int thread_thresh)
 {
-	unsigned long long exp_irq_ts;
+	long long exp_irq_ts;
 	int total;
 	int irq;
 
@@ -562,12 +562,15 @@ static void timerlat_thread_analysis(struct timerlat_aa_data *taa_data, int cpu,
 
 	/*
 	 * Expected IRQ arrival time using the trace clock as the base.
+	 *
+	 * TODO: Add a list of previous IRQ, and then run the list backwards.
 	 */
 	exp_irq_ts = taa_data->timer_irq_start_time - taa_data->timer_irq_start_delay;
-
-	if (exp_irq_ts < taa_data->prev_irq_timstamp + taa_data->prev_irq_duration)
-		printf("  Previous IRQ interference:	\t\t up to  %9.2f us\n",
-			ns_to_usf(taa_data->prev_irq_duration));
+	if (exp_irq_ts < taa_data->prev_irq_timstamp + taa_data->prev_irq_duration) {
+		if (taa_data->prev_irq_timstamp < taa_data->timer_irq_start_time)
+			printf("  Previous IRQ interference:	\t\t up to  %9.2f us\n",
+				ns_to_usf(taa_data->prev_irq_duration));
+	}
 
 	/*
 	 * The delay that the IRQ suffered before starting.
Signature · 1 line
2.38.1