Skip to contentSkip to conversation

perf lock contention: Clear lock addr after use

PATCH Superseded Namhyung Kim 2 participants 3 messages Sep 28, 2023 – Oct 13, 2023

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

PATCH 1/1 perf lock contention: Clear lock addr after use

It checks the current lock to calculated the delta of contention time. The address is saved in the tstamp map which is allocated at begining of contention and released at end of contention.

But it's possible for bpf_map_delete_elem() to fail. In that case, the element in the tstamp map kept for the current lock and it makes the next contention for the same lock tracked incorrectly. Specificially the next contention begin will see the existing element for the task and it'd just return. Then the next contention end will see the element and calculate the time using the timestamp for the previous begin.

This can result in a large value for two small contentions happened from time to time. Let's clear the lock address so that it can be updated next time even if the bpf_map_delete_elem() failed.

---
 tools/perf/util/bpf_skel/lock_contention.bpf.c | 4 ++++
 1 file changed, 4 insertions(+)
tools/perf/util/bpf_skel/lock_contention.bpf.c +4 -0
index 4900a5dfb4a4..b11179452e19 100644
@@ -389,6 +389,7 @@ int contention_end(u64 *ctx)
 
 	duration = bpf_ktime_get_ns() - pelem->timestamp;
 	if ((__s64)duration < 0) {
+		pelem->lock = 0;
 		bpf_map_delete_elem(&tstamp, &pid);
 		__sync_fetch_and_add(&time_fail, 1);
 		return 0;
@@ -422,6 +423,7 @@ int contention_end(u64 *ctx)
 	data = bpf_map_lookup_elem(&lock_stat, &key);
 	if (!data) {
 		if (data_map_full) {
+			pelem->lock = 0;
 			bpf_map_delete_elem(&tstamp, &pid);
 			__sync_fetch_and_add(&data_fail, 1);
 			return 0;
@@ -445,6 +447,7 @@ int contention_end(u64 *ctx)
 				data_map_full = 1;
 			__sync_fetch_and_add(&data_fail, 1);
 		}
+		pelem->lock = 0;
 		bpf_map_delete_elem(&tstamp, &pid);
 		return 0;
 	}
@@ -458,6 +461,7 @@ int contention_end(u64 *ctx)
 	if (data->min_time > duration)
 		data->min_time = duration;
 
+	pelem->lock = 0;
 	bpf_map_delete_elem(&tstamp, &pid);
 	return 0;
 }
Signature · 1 line
2.42.0.582.g8ccd20d70d-goog
#1 lore raw

Em Thu, Sep 28, 2023 at 04:50:18PM -0700, Namhyung Kim escreveu:

Quoted 6 lines

It checks the current lock to calculated the delta of contention time.

The address is saved in the tstamp map which is allocated at begining of contention and released at end of contention.

But it's possible for bpf_map_delete_elem() to fail. In that case, the

How can it fail?

You do:

        pelem = bpf_map_lookup_elem(&tstamp, &pid);
        if (!pelem || pelem->lock != ctx[0])
                return 0;

So it is there, why would the removal using the same key fail?

The patch should work as-is, I'm just curious about what would make there removal of a map entry that was successfully looked up on the same contention_end prog to fail when being removed...

- Arnaldo

Quoted 54 lines

element in the tstamp map kept for the current lock and it makes the next contention for the same lock tracked incorrectly. Specificially the next contention begin will see the existing element for the task and it'd just return. Then the next contention end will see the element and calculate the time using the timestamp for the previous begin.

This can result in a large value for two small contentions happened from time to time. Let's clear the lock address so that it can be updated next time even if the bpf_map_delete_elem() failed.

---
 tools/perf/util/bpf_skel/lock_contention.bpf.c | 4 ++++
 1 file changed, 4 insertions(+)
tools/perf/util/bpf_skel/lock_contention.bpf.c +4 -0
index 4900a5dfb4a4..b11179452e19 100644
@@ -389,6 +389,7 @@ int contention_end(u64 *ctx)
 
 	duration = bpf_ktime_get_ns() - pelem->timestamp;
 	if ((__s64)duration < 0) {
+		pelem->lock = 0;
 		bpf_map_delete_elem(&tstamp, &pid);
 		__sync_fetch_and_add(&time_fail, 1);
 		return 0;
@@ -422,6 +423,7 @@ int contention_end(u64 *ctx)
 	data = bpf_map_lookup_elem(&lock_stat, &key);
 	if (!data) {
 		if (data_map_full) {
+			pelem->lock = 0;
 			bpf_map_delete_elem(&tstamp, &pid);
 			__sync_fetch_and_add(&data_fail, 1);
 			return 0;
@@ -445,6 +447,7 @@ int contention_end(u64 *ctx)
 				data_map_full = 1;
 			__sync_fetch_and_add(&data_fail, 1);
 		}
+		pelem->lock = 0;
 		bpf_map_delete_elem(&tstamp, &pid);
 		return 0;
 	}
@@ -458,6 +461,7 @@ int contention_end(u64 *ctx)
 	if (data->min_time > duration)
 		data->min_time = duration;
 
+	pelem->lock = 0;
 	bpf_map_delete_elem(&tstamp, &pid);
 	return 0;
 }
Signature · 1 line
2.42.0.582.g8ccd20d70d-goog
Signature · 1 line
- Arnaldo
author maintainer
#2 lore raw
Quoted 11 lines · Arnaldo Carvalho de Melo

Em Thu, Sep 28, 2023 at 04:50:18PM -0700, Namhyung Kim escreveu:

Quoted 6 lines

It checks the current lock to calculated the delta of contention time.

The address is saved in the tstamp map which is allocated at begining of contention and released at end of contention.

But it's possible for bpf_map_delete_elem() to fail. In that case, the

How can it fail?

Quoted 7 lines

You do:

        pelem = bpf_map_lookup_elem(&tstamp, &pid);
        if (!pelem || pelem->lock != ctx[0])
                return 0;

So it is there, why would the removal using the same key fail?

It can fail when it doesn't get a lock for the internal bucket. See kernel/bpf/hashtab.c::htab_map_delete_elem().

But I'm not sure whether that's actually possible in this case.

Quoted 4 lines

The patch should work as-is, I'm just curious about what would make there removal of a map entry that was successfully looked up on the same contention_end prog to fail when being removed...

Now I'm seeing some rare error cases like a spinlock wait is longer than a minute. I suspect a bug in this code and try to be more defensive.

Thanks,
Namhyung