Skip to contentSkip to conversation

perf tools: Refactor precise_ip fallback logic

PATCH Applied Zide Chen 4 participants 19 messages Oct 22, 2025 – Feb 6 perf events

lore
Share & tools
lore

Zide Chen

author
#0 lore raw

PATCH 1/1 perf tools: Refactor precise_ip fallback logic

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -3
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
 	if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
 		goto retry_open;
 
+	if (evsel__precise_ip_fallback(evsel))
+		goto retry_open;
+
 	if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
 		goto fallback_missing_features;
 
-	if (evsel__precise_ip_fallback(evsel))
-		goto retry_open;
-
 out_close:
 	if (err)
 		threads->err_thread = thread;
Signature · 1 line
2.51.0
reviewer Acked-by on v1 1/1
#1 lore raw
Quoted 26 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

Quoted 7 lines

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")

Any chance you could help with a test case that covers this? The fallback logic is spread out and easy to introduce subtle bugs into. Just having a test case that does perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls and checks the output for EINVAL when the events are present would be useful, as then we can make sure this doesn't regress on SPR and later. Something with more generic events would of course be better :-)

Thanks,
Ian

Quoted 27 lines
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -4
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
        if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
                goto retry_open;
 
+       if (evsel__precise_ip_fallback(evsel))
+               goto retry_open;
+
        if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
                goto fallback_missing_features;
 
-       if (evsel__precise_ip_fallback(evsel))
-               goto retry_open;
-
 out_close:
        if (err)
                threads->err_thread = thread;
--

2.51.0

author
#2 lore raw
Quoted 37 lines · Ian Rogers
Quoted 26 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

Quoted 7 lines

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")
Quoted 7 lines

Any chance you could help with a test case that covers this? The fallback logic is spread out and easy to introduce subtle bugs into. Just having a test case that does perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls and checks the output for EINVAL when the events are present would be useful, as then we can make sure this doesn't regress on SPR and later. Something with more generic events would of course be better :-)

OK. Maybe a new test "PMU event open fallback tests".

maintainer Applied
#3 lore raw
In reply to Ian Rogers
Quoted 30 lines · Ian Rogers
Quoted 29 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Quoted 6 lines
Quoted 4 lines
Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")

Fell thru the cracks... Still applies cleanly.

Thanks, applied to perf-tools-next,

- Arnaldo

Quoted 38 lines

Any chance you could help with a test case that covers this? The fallback logic is spread out and easy to introduce subtle bugs into. Just having a test case that does perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls and checks the output for EINVAL when the events are present would be useful, as then we can make sure this doesn't regress on SPR and later. Something with more generic events would of course be better :-)

Thanks,
Ian

Quoted 27 lines
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -4
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
        if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
                goto retry_open;
 
+       if (evsel__precise_ip_fallback(evsel))
+               goto retry_open;
+
        if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
                goto fallback_missing_features;
 
-       if (evsel__precise_ip_fallback(evsel))
-               goto retry_open;
-
 out_close:
        if (err)
                threads->err_thread = thread;
--

2.51.0

#4 lore raw
Quoted 36 lines · Arnaldo Carvalho de Melo
Quoted 35 lines · Ian Rogers
Quoted 33 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")
Quoted 5 lines
Quoted 1 line

Fell thru the cracks... Still applies cleanly.

Thanks, applied to perf-tools-next,

So I saw the rest of the thread, removing it...

- Arnaldo

Quoted 40 lines

- Arnaldo

Quoted 38 lines

Any chance you could help with a test case that covers this? The fallback logic is spread out and easy to introduce subtle bugs into. Just having a test case that does perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls and checks the output for EINVAL when the events are present would be useful, as then we can make sure this doesn't regress on SPR and later. Something with more generic events would of course be better :-)

Thanks,
Ian

Quoted 27 lines
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -4
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
        if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
                goto retry_open;
 
+       if (evsel__precise_ip_fallback(evsel))
+               goto retry_open;
+
        if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
                goto fallback_missing_features;
 
-       if (evsel__precise_ip_fallback(evsel))
-               goto retry_open;
-
 out_close:
        if (err)
                threads->err_thread = thread;
--

2.51.0

maintainer
#5 lore raw
In reply to Zide Chen

Hello,

Quoted 8 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

Quoted 6 lines

For example, run the following command on Intel SPR:

$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Quoted 10 lines

kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.

Thanks,
Namhyung

Quoted 35 lines

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -3
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
 	if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
 		goto retry_open;
 
+	if (evsel__precise_ip_fallback(evsel))
+		goto retry_open;
+
 	if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
 		goto fallback_missing_features;
 
-	if (evsel__precise_ip_fallback(evsel))
-		goto retry_open;
-
 out_close:
 	if (err)
 		threads->err_thread = thread;
Signature · 1 line
2.51.0
author
#6 lore raw
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Quoted 9 lines
Quoted 9 lines

kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

Quoted 6 lines
Quoted 2 lines

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.

Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.

evsel__precise_ip_fallback() can restore the precise_ip level after a failed attempt, while evsel__detect_missing_features() cannot recover the event attributes from its failed try.

Therefore, falling back precise_ip first maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if both precise_ip and another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.

Quoted 38 lines

Thanks,
Namhyung

Quoted 35 lines

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -3
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
 	if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
 		goto retry_open;
 
+	if (evsel__precise_ip_fallback(evsel))
+		goto retry_open;
+
 	if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
 		goto fallback_missing_features;
 
-	if (evsel__precise_ip_fallback(evsel))
-		goto retry_open;
-
 out_close:
 	if (err)
 		threads->err_thread = thread;
Signature · 1 line
2.51.0
maintainer
#7 lore raw
Quoted 25 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Quoted 7 lines

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

Quoted 17 lines
Quoted 10 lines
Quoted 10 lines

kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

Quoted 6 lines
Quoted 2 lines

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.

Quoted 7 lines

Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.

evsel__precise_ip_fallback() can restore the precise_ip level after a failed attempt, while evsel__detect_missing_features() cannot recover the event attributes from its failed try.

I think precise_ip_fallback() is just a trial and error for each possible value. While detect_missing_features() checks what the current kernel supports. Trying different precise_ip values with unsupported attributes doesn't make sense.

Thanks,
Namhyung

Quoted 47 lines

Therefore, falling back precise_ip first maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if both precise_ip and another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.

Quoted 38 lines

Thanks,
Namhyung

Quoted 35 lines

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -3
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
 	if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
 		goto retry_open;
 
+	if (evsel__precise_ip_fallback(evsel))
+		goto retry_open;
+
 	if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
 		goto fallback_missing_features;
 
-	if (evsel__precise_ip_fallback(evsel))
-		goto retry_open;
-
 out_close:
 	if (err)
 		threads->err_thread = thread;
Signature · 1 line
2.51.0
author
#8 lore raw
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

Quoted 7 lines
Quoted 4 lines

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").

Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.

Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.

Quoted 21 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines

kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

Quoted 6 lines
Quoted 2 lines

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.

Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.

Quoted 8 lines
Quoted 3 lines

evsel__precise_ip_fallback() can restore the precise_ip level after a failed attempt, while evsel__detect_missing_features() cannot recover the event attributes from its failed try.

I think precise_ip_fallback() is just a trial and error for each possible value. While detect_missing_features() checks what the current kernel supports. Trying different precise_ip values with unsupported attributes doesn't make sense.

When it returns -EINVAL, the cause could be an unsupported precise_ip or something else. We could either end up with "trying different precise_ip values with unsupported attributes", or "trying attributes with unsupported precise_ip".

The perf tool’s fallback logic is a “best effort” mechanism to fix only one issue, not multiple ones. So, IMO, we should place evsel__detect_missing_features() as the last fallback attempt, since it does not restore the event attributes after a failed try.

Quoted 50 lines

Thanks,
Namhyung

Quoted 47 lines

Therefore, falling back precise_ip first maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if both precise_ip and another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.

Quoted 38 lines

Thanks,
Namhyung

Quoted 35 lines

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -3
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
 	if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
 		goto retry_open;
 
+	if (evsel__precise_ip_fallback(evsel))
+		goto retry_open;
+
 	if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
 		goto fallback_missing_features;
 
-	if (evsel__precise_ip_fallback(evsel))
-		goto retry_open;
-
 out_close:
 	if (err)
 		threads->err_thread = thread;
Signature · 1 line
2.51.0
maintainer
#9 lore raw

Hello,

Sorry for the delay.

Quoted 44 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

Quoted 7 lines
Quoted 4 lines

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").

Quoted 6 lines

Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.

Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.

Why does the first succeed and the second fail? Don't they have the same SAMPLE_READ and SAMPLE_TID + inherit flags?

Quoted 37 lines
Quoted 23 lines
Quoted 23 lines
Quoted 12 lines
Quoted 12 lines

kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

Quoted 6 lines
Quoted 2 lines

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.

Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.

Quoted 8 lines
Quoted 3 lines

evsel__precise_ip_fallback() can restore the precise_ip level after a failed attempt, while evsel__detect_missing_features() cannot recover the event attributes from its failed try.

I think precise_ip_fallback() is just a trial and error for each possible value. While detect_missing_features() checks what the current kernel supports. Trying different precise_ip values with unsupported attributes doesn't make sense.

When it returns -EINVAL, the cause could be an unsupported precise_ip or something else. We could either end up with "trying different precise_ip values with unsupported attributes", or "trying attributes with unsupported precise_ip".

Quoted 4 lines

The perf tool’s fallback logic is a “best effort” mechanism to fix only one issue, not multiple ones. So, IMO, we should place evsel__detect_missing_features() as the last fallback attempt, since it does not restore the event attributes after a failed try.

The missing feature check is about the global kernel behavior so there's no point to try if we know the kernel won't support those features. While precise fallback is per-PMU (and per-event?) behavior so it'd be natural to try that after removing must-fail attributes from the missing feature tests.

Thanks,
Namhyung

Quoted 48 lines
Quoted 47 lines
Quoted 47 lines

Therefore, falling back precise_ip first maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if both precise_ip and another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.

Quoted 38 lines

Thanks,
Namhyung

Quoted 35 lines

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -3
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
 	if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
 		goto retry_open;
 
+	if (evsel__precise_ip_fallback(evsel))
+		goto retry_open;
+
 	if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
 		goto fallback_missing_features;
 
-	if (evsel__precise_ip_fallback(evsel))
-		goto retry_open;
-
 out_close:
 	if (err)
 		threads->err_thread = thread;
Signature · 1 line
2.51.0
author
#10 lore raw
Quoted 53 lines · Namhyung Kim

Hello,

Sorry for the delay.

Quoted 48 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

Quoted 7 lines
Quoted 4 lines

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").

Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.

Quoted 5 lines
Quoted 2 lines

Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.

Why does the first succeed and the second fail? Don't they have the same SAMPLE_READ and SAMPLE_TID + inherit flags?

Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (precise_ip == 0).

The second event fails with -EINVAL because, on some platforms, events with precise_ip = 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.

In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).

Quoted 41 lines
Quoted 36 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines

kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.

By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.

Quoted 6 lines
Quoted 2 lines

No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.

I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.

Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.

Quoted 8 lines
Quoted 3 lines

evsel__precise_ip_fallback() can restore the precise_ip level after a failed attempt, while evsel__detect_missing_features() cannot recover the event attributes from its failed try.

I think precise_ip_fallback() is just a trial and error for each possible value. While detect_missing_features() checks what the current kernel supports. Trying different precise_ip values with unsupported attributes doesn't make sense.

When it returns -EINVAL, the cause could be an unsupported precise_ip or something else. We could either end up with "trying different precise_ip values with unsupported attributes", or "trying attributes with unsupported precise_ip".

Quoted 4 lines

The perf tool’s fallback logic is a “best effort” mechanism to fix only one issue, not multiple ones. So, IMO, we should place evsel__detect_missing_features() as the last fallback attempt, since it does not restore the event attributes after a failed try.

Quoted 5 lines

The missing feature check is about the global kernel behavior so there's no point to try if we know the kernel won't support those features. While precise fallback is per-PMU (and per-event?) behavior so it'd be natural to try that after removing must-fail attributes from the missing feature tests.

But someone may argue that since presise_ip is per-event and it's less intrusive, why not try it first?

If we want to keep this principle, we need to ensure that detect missing features does not incorrectly remove valid features, and there’s no need to restore the removed features.

After commit 3b193a57baf1 (“perf tools: Detect missing kernel features properly”), it no longer checks attributes based on the previously failed evsel. Instead, it checks against a dummy event. This makes it difficult to correctly detect features with complex dependencies — for example, group events involves PERF_SAMPLE_READ, PERF_SAMPLE_TID, and inherit.

Another argument is what if the original evsel fails because of multiple invalid attributes? Seems it's hard to trust the missing feature detection to find out "must-fail" attributes.

Quoted 52 lines

Thanks,
Namhyung

Quoted 48 lines
Quoted 47 lines
Quoted 47 lines

Therefore, falling back precise_ip first maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if both precise_ip and another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.

Quoted 38 lines

Thanks,
Namhyung

Quoted 35 lines

This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").

Fixes af954f76eea56453 ("perf tools: Check fallback error and order")
Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")
---
 tools/perf/util/evsel.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -3
index ca74514c8707..6ce32533a213 100644
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
 	if (err == -EMFILE && rlimit__increase_nofile(&set_rlimit))
 		goto retry_open;
 
+	if (evsel__precise_ip_fallback(evsel))
+		goto retry_open;
+
 	if (err == -EINVAL && evsel__detect_missing_features(evsel, cpu))
 		goto fallback_missing_features;
 
-	if (evsel__precise_ip_fallback(evsel))
-		goto retry_open;
-
 out_close:
 	if (err)
 		threads->err_thread = thread;
Signature · 1 line
2.51.0
maintainer
#11 lore raw
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim

Hello,

Sorry for the delay.

Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

Quoted 7 lines
Quoted 4 lines

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").

Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.

Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.

Quoted 6 lines
Quoted 2 lines

Why does the first succeed and the second fail? Don't they have the same SAMPLE_READ and SAMPLE_TID + inherit flags?

Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (precise_ip == 0).

I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and SAMPE_READ. It didn't have precise_ip too.

  $ perf record -e cpu/mem-loads-aux/S -vv true |& less
  ...
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    inherit                          1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1161023  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ...

And it fell back to no-inherit and succeeded. I've also found that it worked even with precise_ip = 3.

  $ perf record -e cpu/mem-loads-aux/PS -vv true |& less
  ...
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    precise_ip                       3         <<<---- here
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8 = 4
  ...

And it works fine on my machine.

  $ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads/PS}' ls
  ...
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.033 MB perf.data (6 samples) ]
Quoted 8 lines

The second event fails with -EINVAL because, on some platforms, events with precise_ip = 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.

In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).

So I don't understand this. Either the first event failed due to inherit set or the second event should succeed with inherit. Maybe there's an unknown bug or something.

Thanks,
namhyung

author
#12 lore raw
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim

Hello,

Sorry for the delay.

Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

Quoted 7 lines
Quoted 4 lines

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").

Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.

Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.

Quoted 6 lines
Quoted 2 lines

Why does the first succeed and the second fail? Don't they have the same SAMPLE_READ and SAMPLE_TID + inherit flags?

Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (precise_ip == 0).

I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and SAMPE_READ. It didn't have precise_ip too.

  $ perf record -e cpu/mem-loads-aux/S -vv true |& less
  ...
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    inherit                          1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1161023  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ...
Quoted 1 line

And it fell back to no-inherit and succeeded.

On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)

It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.

        if (task && group_leader &&
            group_leader->attr.inherit != attr.inherit) {
                err = -EINVAL;
                goto err_task;
        }
Quoted 23 lines

I've also found that it
worked even with precise_ip = 3.

  $ perf record -e cpu/mem-loads-aux/PS -vv true |& less
  ...
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    precise_ip                       3         <<<---- here
Quoted 8 lines
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8 = 4
  ...

Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test, glc_get_event_constraints() successfully forces this event (config == 0x8203) to fixed counter 0, so there’s no issue here.

Quoted 6 lines

And it works fine on my machine.

  $ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads/PS}' ls
  ...
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.033 MB perf.data (6 samples) ]

I don't know why it works for you, but in my tests, this event:

Opening: cpu/mem-loads/PS
------------------------------------------------------------
perf_event_attr:

  type                             4 (cpu)
  size                             248
  config                           0x1cd
(mem_trans_retired.load_latency_gt_1024)
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|READ|ID|PERIOD
  read_format                      ID|GROUP|LOST
  inherit                          1
  freq                             1
  precise_ip                       3
  sample_id_all                    1
  { bp_addr, config1 }             0x3

------------------------------------------------------------

It gets emptyconstraint, then it can't schedule the event on any counter and x86_schedule_events() returns -EINVAL.

glc_get_event_constraints()

{
        struct event_constraint *c;
	
	// It gets the constraint INTEL_PLD_CONSTRAINT(0x1cd, 0xfe)
	// from intel_pebs_constraints(),
        c = icl_get_event_constraints(cpuc, idx, event);

	// When it tries to force :ppp event to fixed counter 0
        if ((event->attr.precise_ip == 3) &&
            !constraint_match(&fixed0_constraint, event->hw.config)) {

		// It happens the constrain doesn't mask fixed counter 0
                if (c->idxmsk64 & BIT_ULL(0)) {
                        return &counter0_constraint;
		
		// It gets here.
                return &emptyconstraint;
        }

        return c;
}

After that, it falls back to non-inherit, and it fails again because the inherit attribute differs from the group leader’s. This carries over to the precise_ip fallback path in the current code.

Quoted 16 lines
Quoted 8 lines

The second event fails with -EINVAL because, on some platforms, events with precise_ip = 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.

In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).

So I don't understand this. Either the first event failed due to inherit set or the second event should succeed with inherit. Maybe there's an unknown bug or something.

Thanks,
namhyung

maintainer
#13 lore raw
Quoted 102 lines · Zide
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim

Hello,

Sorry for the delay.

Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

Quoted 7 lines
Quoted 4 lines

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").

Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.

Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.

Quoted 6 lines
Quoted 2 lines

Why does the first succeed and the second fail? Don't they have the same SAMPLE_READ and SAMPLE_TID + inherit flags?

Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (precise_ip == 0).

I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and SAMPE_READ. It didn't have precise_ip too.

  $ perf record -e cpu/mem-loads-aux/S -vv true |& less
  ...
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    inherit                          1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1161023  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ...
Quoted 5 lines
Quoted 1 line

And it fell back to no-inherit and succeeded.

On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)

Yep, your kernel is recent and has the following commit.

7e8b255650fcfa1d0 ("perf: Support PERF_SAMPLE_READ with inherit")

My kernel is 6.6 and it rejects such a combination. I'll test it on newer kernels later.

Quoted 43 lines

It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.

        if (task && group_leader &&
            group_leader->attr.inherit != attr.inherit) {
                err = -EINVAL;
                goto err_task;
        }
Quoted 23 lines

I've also found that it
worked even with precise_ip = 3.

  $ perf record -e cpu/mem-loads-aux/PS -vv true |& less
  ...
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    precise_ip                       3         <<<---- here
Quoted 8 lines
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8 = 4
  ...
Quoted 3 lines

Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test, glc_get_event_constraints() successfully forces this event (config == 0x8203) to fixed counter 0, so there’s no issue here.

That means your missing_features.inherit_sample_read should not be set. It's strange you have that with the recent kernels.

Can you run these commands and show the output here?

  $ perf record -e task-clock:S  true
  $ perf evlist -v

Thanks,
Namhyung

Quoted 74 lines
Quoted 7 lines

And it works fine on my machine.

  $ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads/PS}' ls
  ...
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.033 MB perf.data (6 samples) ]

I don't know why it works for you, but in my tests, this event:

Opening: cpu/mem-loads/PS
------------------------------------------------------------
perf_event_attr:

  type                             4 (cpu)
  size                             248
  config                           0x1cd
(mem_trans_retired.load_latency_gt_1024)
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|READ|ID|PERIOD
  read_format                      ID|GROUP|LOST
  inherit                          1
  freq                             1
  precise_ip                       3
  sample_id_all                    1
  { bp_addr, config1 }             0x3

------------------------------------------------------------

It gets emptyconstraint, then it can't schedule the event on any counter and x86_schedule_events() returns -EINVAL.

glc_get_event_constraints()

{
        struct event_constraint *c;
	
	// It gets the constraint INTEL_PLD_CONSTRAINT(0x1cd, 0xfe)
	// from intel_pebs_constraints(),
        c = icl_get_event_constraints(cpuc, idx, event);

	// When it tries to force :ppp event to fixed counter 0
        if ((event->attr.precise_ip == 3) &&
            !constraint_match(&fixed0_constraint, event->hw.config)) {

		// It happens the constrain doesn't mask fixed counter 0
                if (c->idxmsk64 & BIT_ULL(0)) {
                        return &counter0_constraint;
		
		// It gets here.
                return &emptyconstraint;
        }

        return c;
}

After that, it falls back to non-inherit, and it fails again because the inherit attribute differs from the group leader’s. This carries over to the precise_ip fallback path in the current code.

Quoted 16 lines
Quoted 8 lines

The second event fails with -EINVAL because, on some platforms, events with precise_ip = 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.

In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).

So I don't understand this. Either the first event failed due to inherit set or the second event should succeed with inherit. Maybe there's an unknown bug or something.

Thanks,
namhyung

author
#14 lore raw
Quoted 163 lines · Namhyung Kim
Quoted 102 lines · Zide
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim

Hello,

Sorry for the delay.

Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

Quoted 7 lines
Quoted 4 lines

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").

Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.

Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.

Quoted 6 lines
Quoted 2 lines

Why does the first succeed and the second fail? Don't they have the same SAMPLE_READ and SAMPLE_TID + inherit flags?

Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (precise_ip == 0).

I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and SAMPE_READ. It didn't have precise_ip too.

  $ perf record -e cpu/mem-loads-aux/S -vv true |& less
  ...
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    inherit                          1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1161023  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ...
Quoted 5 lines
Quoted 1 line

And it fell back to no-inherit and succeeded.

On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)

Yep, your kernel is recent and has the following commit.

7e8b255650fcfa1d0 ("perf: Support PERF_SAMPLE_READ with inherit")

My kernel is 6.6 and it rejects such a combination. I'll test it on newer kernels later.

Quoted 43 lines

It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.

        if (task && group_leader &&
            group_leader->attr.inherit != attr.inherit) {
                err = -EINVAL;
                goto err_task;
        }
Quoted 23 lines

I've also found that it
worked even with precise_ip = 3.

  $ perf record -e cpu/mem-loads-aux/PS -vv true |& less
  ...
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    precise_ip                       3         <<<---- here
Quoted 8 lines
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8 = 4
  ...
Quoted 3 lines

Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test, glc_get_event_constraints() successfully forces this event (config == 0x8203) to fixed counter 0, so there’s no issue here.

Quoted 7 lines

That means your missing_features.inherit_sample_read should not be set. It's strange you have that with the recent kernels.

Can you run these commands and show the output here?

  $ perf record -e task-clock:S  true
  $ perf evlist -v

On 6.18.0-rc4:

$ perf record -e task-clock:S  true
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.006 MB perf.data ]
$ perf evlist -v
task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1
(PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000,
sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled:
1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq:
1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1,
ksymbol: 1, bpf_event: 1, build_id: 1
Quoted 77 lines

Thanks,
Namhyung

Quoted 74 lines
Quoted 7 lines

And it works fine on my machine.

  $ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads/PS}' ls
  ...
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.033 MB perf.data (6 samples) ]

I don't know why it works for you, but in my tests, this event:

Opening: cpu/mem-loads/PS
------------------------------------------------------------
perf_event_attr:

  type                             4 (cpu)
  size                             248
  config                           0x1cd
(mem_trans_retired.load_latency_gt_1024)
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|READ|ID|PERIOD
  read_format                      ID|GROUP|LOST
  inherit                          1
  freq                             1
  precise_ip                       3
  sample_id_all                    1
  { bp_addr, config1 }             0x3

------------------------------------------------------------

It gets emptyconstraint, then it can't schedule the event on any counter and x86_schedule_events() returns -EINVAL.

glc_get_event_constraints()

{
        struct event_constraint *c;
	
	// It gets the constraint INTEL_PLD_CONSTRAINT(0x1cd, 0xfe)
	// from intel_pebs_constraints(),
        c = icl_get_event_constraints(cpuc, idx, event);

	// When it tries to force :ppp event to fixed counter 0
        if ((event->attr.precise_ip == 3) &&
            !constraint_match(&fixed0_constraint, event->hw.config)) {

		// It happens the constrain doesn't mask fixed counter 0
                if (c->idxmsk64 & BIT_ULL(0)) {
                        return &counter0_constraint;
		
		// It gets here.
                return &emptyconstraint;
        }

        return c;
}

After that, it falls back to non-inherit, and it fails again because the inherit attribute differs from the group leader’s. This carries over to the precise_ip fallback path in the current code.

Quoted 16 lines
Quoted 8 lines

The second event fails with -EINVAL because, on some platforms, events with precise_ip = 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.

In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).

So I don't understand this. Either the first event failed due to inherit set or the second event should succeed with inherit. Maybe there's an unknown bug or something.

Thanks,
namhyung

maintainer
#15 lore raw
Quoted 180 lines · Zide
Quoted 163 lines · Namhyung Kim
Quoted 102 lines · Zide
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim

Hello,

Sorry for the delay.

Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

Quoted 7 lines
Quoted 4 lines

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").

Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.

Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.

Quoted 6 lines
Quoted 2 lines

Why does the first succeed and the second fail? Don't they have the same SAMPLE_READ and SAMPLE_TID + inherit flags?

Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (precise_ip == 0).

I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and SAMPE_READ. It didn't have precise_ip too.

  $ perf record -e cpu/mem-loads-aux/S -vv true |& less
  ...
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    inherit                          1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1161023  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ...
Quoted 5 lines
Quoted 1 line

And it fell back to no-inherit and succeeded.

On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)

Yep, your kernel is recent and has the following commit.

7e8b255650fcfa1d0 ("perf: Support PERF_SAMPLE_READ with inherit")

My kernel is 6.6 and it rejects such a combination. I'll test it on newer kernels later.

Quoted 43 lines

It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.

        if (task && group_leader &&
            group_leader->attr.inherit != attr.inherit) {
                err = -EINVAL;
                goto err_task;
        }
Quoted 23 lines

I've also found that it
worked even with precise_ip = 3.

  $ perf record -e cpu/mem-loads-aux/PS -vv true |& less
  ...
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    precise_ip                       3         <<<---- here
Quoted 8 lines
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8 = 4
  ...
Quoted 3 lines

Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test, glc_get_event_constraints() successfully forces this event (config == 0x8203) to fixed counter 0, so there’s no issue here.

Quoted 7 lines

That means your missing_features.inherit_sample_read should not be set. It's strange you have that with the recent kernels.

Can you run these commands and show the output here?

  $ perf record -e task-clock:S  true
  $ perf evlist -v

On 6.18.0-rc4:

$ perf record -e task-clock:S  true
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.006 MB perf.data ]
Quoted 7 lines
$ perf evlist -v
task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1
(PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000,
sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled:
1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq:
1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1,
ksymbol: 1, bpf_event: 1, build_id: 1

Thanks for sharing this. Yep, it has the inherit bit.

I think there's a bug in the missing feature test. Indeed, it should also have PERF_SAMPLE_TID for the test according to the kernel comment.

	/*
	 * We do not support PERF_SAMPLE_READ on inherited events unless
	 * PERF_SAMPLE_TID is also selected, which allows inherited events to
	 * collect per-thread samples.
	 * See perf_output_read().
	 */
	if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPLE_TID))
		return ERR_PTR(-EINVAL);

I'll send a patch soon.

Thanks,
Namhyung

author
#16 lore raw
Quoted 194 lines · Namhyung Kim
Quoted 180 lines · Zide
Quoted 163 lines · Namhyung Kim
Quoted 102 lines · Zide
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim

Hello,

Sorry for the delay.

Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim

Hello,

Quoted 10 lines · Zide Chen

Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.

However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.

For example, run the following command on Intel SPR:

Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls

Opening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a

I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?

Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).

Quoted 7 lines
Quoted 4 lines

When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().

Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?

The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").

Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.

Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.

Quoted 6 lines
Quoted 2 lines

Why does the first succeed and the second fail? Don't they have the same SAMPLE_READ and SAMPLE_TID + inherit flags?

Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (precise_ip == 0).

I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and SAMPE_READ. It didn't have precise_ip too.

  $ perf record -e cpu/mem-loads-aux/S -vv true |& less
  ...
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    inherit                          1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1161023  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ...
Quoted 5 lines
Quoted 1 line

And it fell back to no-inherit and succeeded.

On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)

Yep, your kernel is recent and has the following commit.

7e8b255650fcfa1d0 ("perf: Support PERF_SAMPLE_READ with inherit")

My kernel is 6.6 and it rejects such a combination. I'll test it on newer kernels later.

Quoted 43 lines

It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.

        if (task && group_leader &&
            group_leader->attr.inherit != attr.inherit) {
                err = -EINVAL;
                goto err_task;
        }
Quoted 23 lines

I've also found that it
worked even with precise_ip = 3.

  $ perf record -e cpu/mem-loads-aux/PS -vv true |& less
  ...
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8
  sys_perf_event_open failed, error -22
  Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
  ------------------------------------------------------------
  perf_event_attr:
    type                             4 (cpu)
    size                             136
    config                           0x8203 (mem-loads-aux)
    { sample_period, sample_freq }   4000
    sample_type                      IP|TID|TIME|READ|ID|PERIOD
    read_format                      ID|LOST
    disabled                         1
    mmap                             1
    comm                             1
    freq                             1
    enable_on_exec                   1
    task                             1
    precise_ip                       3         <<<---- here
Quoted 8 lines
    sample_id_all                    1
    mmap2                            1
    comm_exec                        1
    ksymbol                          1
    bpf_event                        1
  ------------------------------------------------------------
  sys_perf_event_open: pid 1172834  cpu 0  group_fd -1  flags 0x8 = 4
  ...
Quoted 3 lines

Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test, glc_get_event_constraints() successfully forces this event (config == 0x8203) to fixed counter 0, so there’s no issue here.

Quoted 7 lines

That means your missing_features.inherit_sample_read should not be set. It's strange you have that with the recent kernels.

Can you run these commands and show the output here?

  $ perf record -e task-clock:S  true
  $ perf evlist -v

On 6.18.0-rc4:

$ perf record -e task-clock:S  true
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.006 MB perf.data ]
Quoted 7 lines
$ perf evlist -v
task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1
(PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000,
sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled:
1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq:
1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1,
ksymbol: 1, bpf_event: 1, build_id: 1

Thanks for sharing this. Yep, it has the inherit bit.

I think there's a bug in the missing feature test. Indeed, it should also have PERF_SAMPLE_TID for the test according to the kernel comment.

Quoted 8 lines
	/*
	 * We do not support PERF_SAMPLE_READ on inherited events unless
	 * PERF_SAMPLE_TID is also selected, which allows inherited events to
	 * collect per-thread samples.
	 * See perf_output_read().
	 */
	if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPLE_TID))
		return ERR_PTR(-EINVAL);

It seems that the purpose of the inherit_sample_read fallback is to remove the inherit attribute when both PERF_SAMPLE_READ and inherit are present, but PERF_SAMPLE_TID is not. The new change may not be able to accomplish this?

Quoted 6 lines

I'll send a patch soon.

Thanks,
Namhyung

maintainer
#17 lore raw
Quoted 45 lines · Zide
Quoted 33 lines · Namhyung Kim
Quoted 19 lines · Zide
Quoted 9 lines · Namhyung Kim
Quoted 3 lines · Zide

On 11/6/2025 10:52 AM, Namhyung Kim wrote:

Can you run these commands and show the output here?

  $ perf record -e task-clock:S  true
  $ perf evlist -v

On 6.18.0-rc4:

$ perf record -e task-clock:S  true
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.006 MB perf.data ]
Quoted 7 lines
$ perf evlist -v
task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1
(PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000,
sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled:
1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq:
1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1,
ksymbol: 1, bpf_event: 1, build_id: 1

Thanks for sharing this. Yep, it has the inherit bit.

I think there's a bug in the missing feature test. Indeed, it should also have PERF_SAMPLE_TID for the test according to the kernel comment.

Quoted 8 lines
	/*
	 * We do not support PERF_SAMPLE_READ on inherited events unless
	 * PERF_SAMPLE_TID is also selected, which allows inherited events to
	 * collect per-thread samples.
	 * See perf_output_read().
	 */
	if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPLE_TID))
		return ERR_PTR(-EINVAL);
Quoted 4 lines

It seems that the purpose of the inherit_sample_read fallback is to remove the inherit attribute when both PERF_SAMPLE_READ and inherit are present, but PERF_SAMPLE_TID is not. The new change may not be able to accomplish this?

No, the purpose of the missing feature check is to detect whether the current kernel supports this feature or not. The correct check should pass both READ and TID together.

Thanks,
Namhyung

author
#18 lore raw
Quoted 46 lines · Namhyung Kim
Quoted 45 lines · Zide
Quoted 41 lines · Namhyung Kim
Quoted 19 lines · Zide
Quoted 9 lines · Namhyung Kim
Quoted 3 lines · Zide

On 11/6/2025 10:52 AM, Namhyung Kim wrote:

Can you run these commands and show the output here?

  $ perf record -e task-clock:S  true
  $ perf evlist -v

On 6.18.0-rc4:

$ perf record -e task-clock:S  true
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.006 MB perf.data ]
Quoted 7 lines
$ perf evlist -v
task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1
(PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000,
sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled:
1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq:
1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1,
ksymbol: 1, bpf_event: 1, build_id: 1

Thanks for sharing this. Yep, it has the inherit bit.

I think there's a bug in the missing feature test. Indeed, it should also have PERF_SAMPLE_TID for the test according to the kernel comment.

	/*
	 * We do not support PERF_SAMPLE_READ on inherited events unless
	 * PERF_SAMPLE_TID is also selected, which allows inherited events to
	 * collect per-thread samples.
	 * See perf_output_read().
	 */
	if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPLE_TID))
		return ERR_PTR(-EINVAL);
Quoted 8 lines
Quoted 4 lines

It seems that the purpose of the inherit_sample_read fallback is to remove the inherit attribute when both PERF_SAMPLE_READ and inherit are present, but PERF_SAMPLE_TID is not. The new change may not be able to accomplish this?

No, the purpose of the missing feature check is to detect whether the current kernel supports this feature or not. The correct check should pass both READ and TID together.

OK, thanks!

Quoted 3 lines

Thanks,
Namhyung