perf tools: Refactor precise_ip fallback logic
PATCH 1/1 perf tools: Refactor precise_ip fallback logic
Commit c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic") unconditionally called the precise_ip fallback and moved it after the missing-feature checks so that it could handle EINVAL as well.
However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent precise_ip fallback useless since it will always fail.
For example, run the following command on Intel SPR:
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when precise_ip == 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after precise_ip is reduced.
By moving the precise_ip fallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commit c33aea446bf555ab.
No negative side effects are expected, because the precise_ip level is restored by evsel__precise_ip_fallback() if the fallback does not help.
This also aligns with commit 2b70702917337a8d ("perf tools: Remove evsel__handle_error_quirks()").
--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/util/evsel.c +3 -3
@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread;
Signature · 1 line
2.51.0
Quoted 26 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even afterprecise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.
Quoted 7 lines
This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>
Any chance you could help with a test case that covers this? The fallback logic is spread out and easy to introduce subtle bugs into. Just having a test case that does perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- ls and checks the output for EINVAL when the events are present would be useful, as then we can make sure this doesn't regress on SPR and later. Something with more generic events would of course be better :-)
Thanks,
Ian
Quoted 27 lines
--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/perf/util/evsel.c +3
-4@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread; --2.51.0
Quoted 37 lines · Ian Rogers
Quoted 26 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even afterprecise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.Quoted 7 lines
This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>Acked-by Ian Rogers <[email protected]>
Quoted 7 lines
Any chance you could help with a test case that covers this? The fallback logic is spread out and easy to introduce subtle bugs into. Just having a test case that does
perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsand checks the output for EINVAL when the events are present would be useful, as then we can make sure this doesn't regress on SPR and later. Something with more generic events would of course be better :-)
OK. Maybe a new test "PMU event open fallback tests".
Quoted 30 lines · Ian Rogers
Quoted 29 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even afterprecise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").
Quoted 6 lines
Quoted 4 lines
Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>Acked-by Ian Rogers <[email protected]>
Fell thru the cracks... Still applies cleanly.
Thanks, applied to perf-tools-next,
- Arnaldo
Quoted 38 lines
Any chance you could help with a test case that covers this? The fallback logic is spread out and easy to introduce subtle bugs into. Just having a test case that does
perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsand checks the output for EINVAL when the events are present would be useful, as then we can make sure this doesn't regress on SPR and later. Something with more generic events would of course be better :-)Thanks,
IanQuoted 27 lines
--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/perf/util/evsel.c +3
-4@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread; --2.51.0
Arnaldo Carvalho de Melo
maintainerQuoted 36 lines · Arnaldo Carvalho de Melo
Quoted 35 lines · Ian Rogers
Quoted 33 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers a kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even afterprecise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>
Quoted 5 lines
Quoted 1 line
Acked-by Ian Rogers <[email protected]>Fell thru the cracks... Still applies cleanly.
Thanks, applied to perf-tools-next,
So I saw the rest of the thread, removing it...
- Arnaldo
Quoted 40 lines
- Arnaldo
Quoted 38 lines
Any chance you could help with a test case that covers this? The fallback logic is spread out and easy to introduce subtle bugs into. Just having a test case that does
perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsand checks the output for EINVAL when the events are present would be useful, as then we can make sure this doesn't regress on SPR and later. Something with more generic events would of course be better :-)Thanks,
IanQuoted 27 lines
--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/perf/util/evsel.c +3
-4@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread; --2.51.0
Namhyung Kim
maintainerHello,
Quoted 8 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.
Quoted 6 lines
For example, run the following command on Intel SPR:
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers a
I'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Quoted 10 lines
kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after
precise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.
I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.
Thanks,
Namhyung
Quoted 35 lines
This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/perf/util/evsel.c +3
-3@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread;Signature · 1 line
2.51.0
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
When the second event fails with EINVAL, the current logic calls evsel__detect_missing_features() first. Since this is a PERF_SAMPLE_READ event, the inherit attribute falls back to false, according to the fallback order implemented in evsel__detect_missing_features().
Quoted 9 lines
Quoted 9 lines
kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after
precise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.
Quoted 6 lines
Quoted 2 lines
No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.
Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.
evsel__precise_ip_fallback() can restore the precise_ip level after a failed attempt, while evsel__detect_missing_features() cannot recover the event attributes from its failed try.
Therefore, falling back precise_ip first maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if both precise_ip and another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.
Quoted 38 lines
Thanks,
NamhyungQuoted 35 lines
This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/perf/util/evsel.c +3
-3@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread;Signature · 1 line
2.51.0
Namhyung Kim
maintainerQuoted 25 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Quoted 7 lines
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().
Right, that means the kernel doesn't support PERF_SAMPLE_READ with inherit = true. How did the first event succeed to open then?
Quoted 17 lines
Quoted 10 lines
Quoted 10 lines
kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after
precise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.Quoted 6 lines
Quoted 2 lines
No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.
Quoted 7 lines
Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.
evsel__precise_ip_fallback()can restore theprecise_iplevel after a failed attempt, whileevsel__detect_missing_features()cannot recover the event attributes from its failed try.
I think precise_ip_fallback() is just a trial and error for each possible value. While detect_missing_features() checks what the current kernel supports. Trying different precise_ip values with unsupported attributes doesn't make sense.
Thanks,
Namhyung
Quoted 47 lines
Therefore, falling back
precise_ipfirst maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if bothprecise_ipand another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.Quoted 38 lines
Thanks,
NamhyungQuoted 35 lines
This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/perf/util/evsel.c +3
-3@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread;Signature · 1 line
2.51.0
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
Quoted 7 lines
Quoted 4 lines
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().Right, that means the kernel doesn't support
PERF_SAMPLE_READwith inherit = true. How did the first event succeed to open then?
The perf tool sets PERF_SAMPLE_TID for Inherit + PERF_SAMPLE_READ events, as implemented in commit 90035d3cd876 ("tools/perf: Allow inherit + PERF_SAMPLE_READ when opening event").
Meanwhile, commit 7e8b255650fc ("perf: Support PERF_SAMPLE_READ with inherit") rejects a perf event if has_inherit_and_sample_read(attr) is true and PERF_SAMPLE_TID is not set in attr->sample_type.
Therefore, the first event succeeded, while the one opened in evsel__detect_missing_features() which doesn't have PERF_SAMPLE_TID failed.
Quoted 21 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after
precise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.Quoted 6 lines
Quoted 2 lines
No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.
Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.
Quoted 8 lines
Quoted 3 lines
evsel__precise_ip_fallback()can restore theprecise_iplevel after a failed attempt, whileevsel__detect_missing_features()cannot recover the event attributes from its failed try.I think
precise_ip_fallback()is just a trial and error for each possible value. Whiledetect_missing_features()checks what the current kernel supports. Trying differentprecise_ipvalues with unsupported attributes doesn't make sense.
When it returns -EINVAL, the cause could be an unsupported precise_ip or something else. We could either end up with "trying different precise_ip values with unsupported attributes", or "trying attributes with unsupported precise_ip".
The perf tool’s fallback logic is a “best effort” mechanism to fix only one issue, not multiple ones. So, IMO, we should place evsel__detect_missing_features() as the last fallback attempt, since it does not restore the event attributes after a failed try.
Quoted 50 lines
Thanks,
NamhyungQuoted 47 lines
Therefore, falling back
precise_ipfirst maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if bothprecise_ipand another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.Quoted 38 lines
Thanks,
NamhyungQuoted 35 lines
This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/perf/util/evsel.c +3
-3@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread;Signature · 1 line
2.51.0
Namhyung Kim
maintainerHello,
Sorry for the delay.
Quoted 44 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
Quoted 7 lines
Quoted 4 lines
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().Right, that means the kernel doesn't support
PERF_SAMPLE_READwith inherit = true. How did the first event succeed to open then?The perf tool sets
PERF_SAMPLE_TIDfor Inherit +PERF_SAMPLE_READevents, as implemented in commit90035d3cd876("tools/perf: Allow inherit +PERF_SAMPLE_READwhen opening event").
Quoted 6 lines
Meanwhile, commit
7e8b255650fc("perf: SupportPERF_SAMPLE_READwith inherit") rejects a perf event ifhas_inherit_and_sample_read(attr)is true andPERF_SAMPLE_TIDis not set inattr->sample_type.Therefore, the first event succeeded, while the one opened in
evsel__detect_missing_features()which doesn't havePERF_SAMPLE_TIDfailed.
Why does the first succeed and the second fail? Don't they have the same SAMPLE_READ and SAMPLE_TID + inherit flags?
Quoted 37 lines
Quoted 23 lines
Quoted 23 lines
Quoted 12 lines
Quoted 12 lines
kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after
precise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.Quoted 6 lines
Quoted 2 lines
No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.
Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.
Quoted 8 lines
Quoted 3 lines
evsel__precise_ip_fallback()can restore theprecise_iplevel after a failed attempt, whileevsel__detect_missing_features()cannot recover the event attributes from its failed try.I think
precise_ip_fallback()is just a trial and error for each possible value. Whiledetect_missing_features()checks what the current kernel supports. Trying differentprecise_ipvalues with unsupported attributes doesn't make sense.When it returns
-EINVAL, the cause could be an unsupportedprecise_ipor something else. We could either end up with "trying differentprecise_ipvalues with unsupported attributes", or "trying attributes with unsupportedprecise_ip".
Quoted 4 lines
The perf tool’s fallback logic is a “best effort” mechanism to fix only one issue, not multiple ones. So, IMO, we should place
evsel__detect_missing_features()as the last fallback attempt, since it does not restore the event attributes after a failed try.
The missing feature check is about the global kernel behavior so there's no point to try if we know the kernel won't support those features. While precise fallback is per-PMU (and per-event?) behavior so it'd be natural to try that after removing must-fail attributes from the missing feature tests.
Thanks,
Namhyung
Quoted 48 lines
Quoted 47 lines
Quoted 47 lines
Therefore, falling back
precise_ipfirst maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if bothprecise_ipand another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.Quoted 38 lines
Thanks,
NamhyungQuoted 35 lines
This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/perf/util/evsel.c +3
-3@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread;Signature · 1 line
2.51.0
Quoted 53 lines · Namhyung Kim
Hello,
Sorry for the delay.
Quoted 48 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
Quoted 7 lines
Quoted 4 lines
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().Right, that means the kernel doesn't support
PERF_SAMPLE_READwith inherit = true. How did the first event succeed to open then?The perf tool sets
PERF_SAMPLE_TIDfor Inherit +PERF_SAMPLE_READevents, as implemented in commit90035d3cd876("tools/perf: Allow inherit +PERF_SAMPLE_READwhen opening event").Meanwhile, commit
7e8b255650fc("perf: SupportPERF_SAMPLE_READwith inherit") rejects a perf event ifhas_inherit_and_sample_read(attr)is true andPERF_SAMPLE_TIDis not set inattr->sample_type.
Quoted 5 lines
Quoted 2 lines
Therefore, the first event succeeded, while the one opened in
evsel__detect_missing_features()which doesn't havePERF_SAMPLE_TIDfailed.Why does the first succeed and the second fail? Don't they have the same
SAMPLE_READandSAMPLE_TID+ inherit flags?
Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (precise_ip == 0).
The second event fails with -EINVAL because, on some platforms, events with precise_ip = 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.
In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).
Quoted 41 lines
Quoted 36 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines
kernel check failure since it doesn't match the group leader's inherit attribute. As a result, it continues to fail even after
precise_ipis reduced.By moving the
precise_ipfallback earlier, this issue is resolved, as well as the kernel test robot report mentioned in commitc33aea446bf555ab.Quoted 6 lines
Quoted 2 lines
No negative side effects are expected, because the
precise_iplevel is restored byevsel__precise_ip_fallback()if the fallback does not help.I'm not sure.. some events may need a different (i.e. lower) precise level than the max. I think checking the missing feature later will use the max precise level always.
Yes, but seems the basic idea of the event open fallback logic is to check whether it's lucky enough to open the event by falling back one single attribute, not multiple attributes.
Quoted 8 lines
Quoted 3 lines
evsel__precise_ip_fallback()can restore theprecise_iplevel after a failed attempt, whileevsel__detect_missing_features()cannot recover the event attributes from its failed try.I think
precise_ip_fallback()is just a trial and error for each possible value. Whiledetect_missing_features()checks what the current kernel supports. Trying differentprecise_ipvalues with unsupported attributes doesn't make sense.When it returns
-EINVAL, the cause could be an unsupportedprecise_ipor something else. We could either end up with "trying differentprecise_ipvalues with unsupported attributes", or "trying attributes with unsupportedprecise_ip".Quoted 4 lines
The perf tool’s fallback logic is a “best effort” mechanism to fix only one issue, not multiple ones. So, IMO, we should place
evsel__detect_missing_features()as the last fallback attempt, since it does not restore the event attributes after a failed try.
Quoted 5 lines
The missing feature check is about the global kernel behavior so there's no point to try if we know the kernel won't support those features. While precise fallback is per-PMU (and per-event?) behavior so it'd be natural to try that after removing must-fail attributes from the missing feature tests.
But someone may argue that since presise_ip is per-event and it's less intrusive, why not try it first?
If we want to keep this principle, we need to ensure that detect missing features does not incorrectly remove valid features, and there’s no need to restore the removed features.
After commit 3b193a57baf1 (“perf tools: Detect missing kernel features properly”), it no longer checks attributes based on the previously failed evsel. Instead, it checks against a dummy event. This makes it difficult to correctly detect features with complex dependencies — for example, group events involves PERF_SAMPLE_READ, PERF_SAMPLE_TID, and inherit.
Another argument is what if the original evsel fails because of multiple invalid attributes? Seems it's hard to trust the missing feature detection to find out "must-fail" attributes.
Quoted 52 lines
Thanks,
NamhyungQuoted 48 lines
Quoted 47 lines
Quoted 47 lines
Therefore, falling back
precise_ipfirst maintains the intended “one-by-one” fallback logic. If it’s placed later, it may combine two fallbacks, which can cause failures like the example above. Of course, in theory, there might be cases where an event can be opened if bothprecise_ipand another feature are relaxed together, but I haven’t exhaustively checked whether such cases actually exist.Quoted 38 lines
Thanks,
NamhyungQuoted 35 lines
This also aligns with commit
2b70702917337a8d("perf tools: Removeevsel__handle_error_quirks()").Fixes af954f76eea56453 ("perf tools: Check fallback error and order")Fixes c33aea446bf555ab ("perf tools: Fix precise_ip fallback logic")Reviewed-by Dapeng Mi <[email protected]>Signed-off-by Zide Chen <[email protected]>--- tools/perf/util/evsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)tools/perf/util/evsel.c +3
-3@@ -2714,12 +2714,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,if (err == -EMFILE &&rlimit__increase_nofile (&set_rlimit)) goto retry_open; +if (evsel__precise_ip_fallback (evsel)) + goto retry_open; +if (err == -EINVAL &&evsel__detect_missing_features (evsel, cpu)) goto fallback_missing_features; -if (evsel__precise_ip_fallback (evsel)) - goto retry_open; - out_close:if (err) threads->err_thread = thread;Signature · 1 line
2.51.0
Namhyung Kim
maintainerQuoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim
Hello,
Sorry for the delay.
Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
Quoted 7 lines
Quoted 4 lines
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().Right, that means the kernel doesn't support
PERF_SAMPLE_READwith inherit = true. How did the first event succeed to open then?The perf tool sets
PERF_SAMPLE_TIDfor Inherit +PERF_SAMPLE_READevents, as implemented in commit90035d3cd876("tools/perf: Allow inherit +PERF_SAMPLE_READwhen opening event").Meanwhile, commit
7e8b255650fc("perf: SupportPERF_SAMPLE_READwith inherit") rejects a perf event ifhas_inherit_and_sample_read(attr)is true andPERF_SAMPLE_TIDis not set inattr->sample_type.Therefore, the first event succeeded, while the one opened in
evsel__detect_missing_features()which doesn't havePERF_SAMPLE_TIDfailed.
Quoted 6 lines
Quoted 2 lines
Why does the first succeed and the second fail? Don't they have the same
SAMPLE_READandSAMPLE_TID+ inherit flags?Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (
precise_ip== 0).
I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and SAMPE_READ. It didn't have precise_ip too.
$ perf record -e cpu/mem-loads-aux/S -vv true |& less
...
------------------------------------------------------------
perf_event_attr:
type 4 (cpu)
size 136
config 0x8203 (mem-loads-aux)
{ sample_period, sample_freq } 4000
sample_type IP|TID|TIME|READ|ID|PERIOD
read_format ID|LOST
disabled 1
inherit 1
mmap 1
comm 1
freq 1
enable_on_exec 1
task 1
sample_id_all 1
mmap2 1
comm_exec 1
ksymbol 1
bpf_event 1
------------------------------------------------------------
sys_perf_event_open: pid 1161023 cpu 0 group_fd -1 flags 0x8
sys_perf_event_open failed, error -22
Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
...And it fell back to no-inherit and succeeded. I've also found that it worked even with precise_ip = 3.
$ perf record -e cpu/mem-loads-aux/PS -vv true |& less
...
sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8
sys_perf_event_open failed, error -22
Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit.
------------------------------------------------------------
perf_event_attr:
type 4 (cpu)
size 136
config 0x8203 (mem-loads-aux)
{ sample_period, sample_freq } 4000
sample_type IP|TID|TIME|READ|ID|PERIOD
read_format ID|LOST
disabled 1
mmap 1
comm 1
freq 1
enable_on_exec 1
task 1
precise_ip 3 <<<---- here
sample_id_all 1
mmap2 1
comm_exec 1
ksymbol 1
bpf_event 1
------------------------------------------------------------
sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 = 4
...And it works fine on my machine.
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads/PS}' ls
...
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.033 MB perf.data (6 samples) ]Quoted 8 lines
The second event fails with
-EINVALbecause, on some platforms, events withprecise_ip= 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).
So I don't understand this. Either the first event failed due to inherit set or the second event should succeed with inherit. Maybe there's an unknown bug or something.
Thanks,
namhyung
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim
Hello,
Sorry for the delay.
Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
Quoted 7 lines
Quoted 4 lines
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().Right, that means the kernel doesn't support
PERF_SAMPLE_READwith inherit = true. How did the first event succeed to open then?The perf tool sets
PERF_SAMPLE_TIDfor Inherit +PERF_SAMPLE_READevents, as implemented in commit90035d3cd876("tools/perf: Allow inherit +PERF_SAMPLE_READwhen opening event").Meanwhile, commit
7e8b255650fc("perf: SupportPERF_SAMPLE_READwith inherit") rejects a perf event ifhas_inherit_and_sample_read(attr)is true andPERF_SAMPLE_TIDis not set inattr->sample_type.Therefore, the first event succeeded, while the one opened in
evsel__detect_missing_features()which doesn't havePERF_SAMPLE_TIDfailed.Quoted 6 lines
Quoted 2 lines
Why does the first succeed and the second fail? Don't they have the same
SAMPLE_READandSAMPLE_TID+ inherit flags?Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (
precise_ip== 0).I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and
SAMPE_READ. It didn't haveprecise_iptoo.$ perf record -e cpu/mem-loads-aux/S -vv true |& less ... ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 inherit 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 ------------------------------------------------------------ sys_perf_event_open: pid 1161023 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ...
Quoted 1 line
And it fell back to no-inherit and succeeded.
On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)
It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.
if (task && group_leader &&
group_leader->attr.inherit != attr.inherit) {
err = -EINVAL;
goto err_task;
}Quoted 23 lines
I've also found that it
worked even withprecise_ip= 3.$ perf record -e cpu/mem-loads-aux/PS -vv true |& less ... sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 precise_ip 3 <<<---- here
Quoted 8 lines
sample_id_all 1
mmap2 1
comm_exec 1
ksymbol 1
bpf_event 1
------------------------------------------------------------
sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 = 4
...Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test, glc_get_event_constraints() successfully forces this event (config == 0x8203) to fixed counter 0, so there’s no issue here.
Quoted 6 lines
And it works fine on my machine.
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads/PS}' ls ... [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.033 MB perf.data (6 samples) ]
I don't know why it works for you, but in my tests, this event:
Opening: cpu/mem-loads/PS
------------------------------------------------------------perf_event_attr:
type 4 (cpu)
size 248
config 0x1cd
(mem_trans_retired.load_latency_gt_1024)
{ sample_period, sample_freq } 4000
sample_type IP|TID|TIME|READ|ID|PERIOD
read_format ID|GROUP|LOST
inherit 1
freq 1
precise_ip 3
sample_id_all 1
{ bp_addr, config1 } 0x3------------------------------------------------------------
It gets emptyconstraint, then it can't schedule the event on any counter and x86_schedule_events() returns -EINVAL.
glc_get_event_constraints()
{
struct event_constraint *c;
// It gets the constraint INTEL_PLD_CONSTRAINT(0x1cd, 0xfe)
// from intel_pebs_constraints(),
c = icl_get_event_constraints(cpuc, idx, event);
// When it tries to force :ppp event to fixed counter 0
if ((event->attr.precise_ip == 3) &&
!constraint_match(&fixed0_constraint, event->hw.config)) {
// It happens the constrain doesn't mask fixed counter 0
if (c->idxmsk64 & BIT_ULL(0)) {
return &counter0_constraint;
// It gets here.
return &emptyconstraint;
}
return c;
}After that, it falls back to non-inherit, and it fails again because the inherit attribute differs from the group leader’s. This carries over to the precise_ip fallback path in the current code.
Quoted 16 lines
Quoted 8 lines
The second event fails with
-EINVALbecause, on some platforms, events withprecise_ip= 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).
So I don't understand this. Either the first event failed due to inherit set or the second event should succeed with inherit. Maybe there's an unknown bug or something.
Thanks,
namhyung
Namhyung Kim
maintainerQuoted 102 lines · Zide
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim
Hello,
Sorry for the delay.
Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
Quoted 7 lines
Quoted 4 lines
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().Right, that means the kernel doesn't support
PERF_SAMPLE_READwith inherit = true. How did the first event succeed to open then?The perf tool sets
PERF_SAMPLE_TIDfor Inherit +PERF_SAMPLE_READevents, as implemented in commit90035d3cd876("tools/perf: Allow inherit +PERF_SAMPLE_READwhen opening event").Meanwhile, commit
7e8b255650fc("perf: SupportPERF_SAMPLE_READwith inherit") rejects a perf event ifhas_inherit_and_sample_read(attr)is true andPERF_SAMPLE_TIDis not set inattr->sample_type.Therefore, the first event succeeded, while the one opened in
evsel__detect_missing_features()which doesn't havePERF_SAMPLE_TIDfailed.Quoted 6 lines
Quoted 2 lines
Why does the first succeed and the second fail? Don't they have the same
SAMPLE_READandSAMPLE_TID+ inherit flags?Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (
precise_ip== 0).I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and
SAMPE_READ. It didn't haveprecise_iptoo.$ perf record -e cpu/mem-loads-aux/S -vv true |& less ... ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 inherit 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 ------------------------------------------------------------ sys_perf_event_open: pid 1161023 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ...
Quoted 5 lines
Quoted 1 line
And it fell back to no-inherit and succeeded.
On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)
Yep, your kernel is recent and has the following commit.
7e8b255650fcfa1d0 ("perf: Support PERF_SAMPLE_READ with inherit")
My kernel is 6.6 and it rejects such a combination. I'll test it on newer kernels later.
Quoted 43 lines
It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.
if (task && group_leader && group_leader->attr.inherit != attr.inherit) { err = -EINVAL; goto err_task; }Quoted 23 lines
I've also found that it
worked even withprecise_ip= 3.$ perf record -e cpu/mem-loads-aux/PS -vv true |& less ... sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 precise_ip 3 <<<---- hereQuoted 8 lines
sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 ------------------------------------------------------------ sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 = 4 ...
Quoted 3 lines
Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test,
glc_get_event_constraints()successfully forces this event (config ==0x8203) to fixed counter 0, so there’s no issue here.
That means your missing_features.inherit_sample_read should not be set. It's strange you have that with the recent kernels.
Can you run these commands and show the output here?
$ perf record -e task-clock:S true $ perf evlist -v
Thanks,
Namhyung
Quoted 74 lines
Quoted 7 lines
And it works fine on my machine.
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads/PS}' ls ... [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.033 MB perf.data (6 samples) ]I don't know why it works for you, but in my tests, this event:
Opening: cpu/mem-loads/PS
------------------------------------------------------------perf_event_attr:type 4 (cpu) size 248 config 0x1cd (mem_trans_retired.load_latency_gt_1024) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|GROUP|LOST inherit 1 freq 1 precise_ip 3 sample_id_all 1 { bp_addr, config1 } 0x3------------------------------------------------------------
It gets emptyconstraint, then it can't schedule the event on any counter and
x86_schedule_events()returns-EINVAL.
glc_get_event_constraints(){ struct event_constraint *c; // It gets the constraint INTEL_PLD_CONSTRAINT(0x1cd, 0xfe) // from intel_pebs_constraints(), c = icl_get_event_constraints(cpuc, idx, event); // When it tries to force :ppp event to fixed counter 0 if ((event->attr.precise_ip == 3) && !constraint_match(&fixed0_constraint, event->hw.config)) { // It happens the constrain doesn't mask fixed counter 0 if (c->idxmsk64 & BIT_ULL(0)) { return &counter0_constraint; // It gets here. return &emptyconstraint; } return c; }After that, it falls back to non-inherit, and it fails again because the inherit attribute differs from the group leader’s. This carries over to the
precise_ipfallback path in the current code.Quoted 16 lines
Quoted 8 lines
The second event fails with
-EINVALbecause, on some platforms, events withprecise_ip= 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).
So I don't understand this. Either the first event failed due to inherit set or the second event should succeed with inherit. Maybe there's an unknown bug or something.
Thanks,
namhyung
Quoted 163 lines · Namhyung Kim
Quoted 102 lines · Zide
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim
Hello,
Sorry for the delay.
Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
Quoted 7 lines
Quoted 4 lines
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().Right, that means the kernel doesn't support
PERF_SAMPLE_READwith inherit = true. How did the first event succeed to open then?The perf tool sets
PERF_SAMPLE_TIDfor Inherit +PERF_SAMPLE_READevents, as implemented in commit90035d3cd876("tools/perf: Allow inherit +PERF_SAMPLE_READwhen opening event").Meanwhile, commit
7e8b255650fc("perf: SupportPERF_SAMPLE_READwith inherit") rejects a perf event ifhas_inherit_and_sample_read(attr)is true andPERF_SAMPLE_TIDis not set inattr->sample_type.Therefore, the first event succeeded, while the one opened in
evsel__detect_missing_features()which doesn't havePERF_SAMPLE_TIDfailed.Quoted 6 lines
Quoted 2 lines
Why does the first succeed and the second fail? Don't they have the same
SAMPLE_READandSAMPLE_TID+ inherit flags?Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (
precise_ip== 0).I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and
SAMPE_READ. It didn't haveprecise_iptoo.$ perf record -e cpu/mem-loads-aux/S -vv true |& less ... ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 inherit 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 ------------------------------------------------------------ sys_perf_event_open: pid 1161023 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ...Quoted 5 lines
Quoted 1 line
And it fell back to no-inherit and succeeded.
On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)
Yep, your kernel is recent and has the following commit.
7e8b255650fcfa1d0("perf: SupportPERF_SAMPLE_READwith inherit")My kernel is 6.6 and it rejects such a combination. I'll test it on newer kernels later.
Quoted 43 lines
It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.
if (task && group_leader && group_leader->attr.inherit != attr.inherit) { err = -EINVAL; goto err_task; }Quoted 23 lines
I've also found that it
worked even withprecise_ip= 3.$ perf record -e cpu/mem-loads-aux/PS -vv true |& less ... sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 precise_ip 3 <<<---- hereQuoted 8 lines
sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 ------------------------------------------------------------ sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 = 4 ...Quoted 3 lines
Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test,
glc_get_event_constraints()successfully forces this event (config ==0x8203) to fixed counter 0, so there’s no issue here.
Quoted 7 lines
That means your missing_features.inherit_sample_read should not be set. It's strange you have that with the recent kernels.
Can you run these commands and show the output here?
$ perf record -e task-clock:S true $ perf evlist -v
On 6.18.0-rc4:
$ perf record -e task-clock:S true [ perf record: Woken up 2 times to write data ] [ perf record: Captured and wrote 0.006 MB perf.data ]
$ perf evlist -v
task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1
(PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000,
sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled:
1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq:
1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1,
ksymbol: 1, bpf_event: 1, build_id: 1Quoted 77 lines
Thanks,
NamhyungQuoted 74 lines
Quoted 7 lines
And it works fine on my machine.
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads/PS}' ls ... [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.033 MB perf.data (6 samples) ]I don't know why it works for you, but in my tests, this event:
Opening: cpu/mem-loads/PS
------------------------------------------------------------perf_event_attr:type 4 (cpu) size 248 config 0x1cd (mem_trans_retired.load_latency_gt_1024) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|GROUP|LOST inherit 1 freq 1 precise_ip 3 sample_id_all 1 { bp_addr, config1 } 0x3------------------------------------------------------------
It gets emptyconstraint, then it can't schedule the event on any counter and
x86_schedule_events()returns-EINVAL.
glc_get_event_constraints(){ struct event_constraint *c; // It gets the constraint INTEL_PLD_CONSTRAINT(0x1cd, 0xfe) // from intel_pebs_constraints(), c = icl_get_event_constraints(cpuc, idx, event); // When it tries to force :ppp event to fixed counter 0 if ((event->attr.precise_ip == 3) && !constraint_match(&fixed0_constraint, event->hw.config)) { // It happens the constrain doesn't mask fixed counter 0 if (c->idxmsk64 & BIT_ULL(0)) { return &counter0_constraint; // It gets here. return &emptyconstraint; } return c; }After that, it falls back to non-inherit, and it fails again because the inherit attribute differs from the group leader’s. This carries over to the
precise_ipfallback path in the current code.Quoted 16 lines
Quoted 8 lines
The second event fails with
-EINVALbecause, on some platforms, events withprecise_ip= 3 must be scheduled on fixed counter 0, and it fails if it happens that this counter is unavailable.In the current code, the first fallback attempt (inherit = 0) also fails because the inherit attribute differs from that of the group leader (first event).
So I don't understand this. Either the first event failed due to inherit set or the second event should succeed with inherit. Maybe there's an unknown bug or something.
Thanks,
namhyung
Namhyung Kim
maintainerQuoted 180 lines · Zide
Quoted 163 lines · Namhyung Kim
Quoted 102 lines · Zide
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim
Hello,
Sorry for the delay.
Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
Quoted 7 lines
Quoted 4 lines
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().Right, that means the kernel doesn't support
PERF_SAMPLE_READwith inherit = true. How did the first event succeed to open then?The perf tool sets
PERF_SAMPLE_TIDfor Inherit +PERF_SAMPLE_READevents, as implemented in commit90035d3cd876("tools/perf: Allow inherit +PERF_SAMPLE_READwhen opening event").Meanwhile, commit
7e8b255650fc("perf: SupportPERF_SAMPLE_READwith inherit") rejects a perf event ifhas_inherit_and_sample_read(attr)is true andPERF_SAMPLE_TIDis not set inattr->sample_type.Therefore, the first event succeeded, while the one opened in
evsel__detect_missing_features()which doesn't havePERF_SAMPLE_TIDfailed.Quoted 6 lines
Quoted 2 lines
Why does the first succeed and the second fail? Don't they have the same
SAMPLE_READandSAMPLE_TID+ inherit flags?Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (
precise_ip== 0).I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and
SAMPE_READ. It didn't haveprecise_iptoo.$ perf record -e cpu/mem-loads-aux/S -vv true |& less ... ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 inherit 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 ------------------------------------------------------------ sys_perf_event_open: pid 1161023 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ...Quoted 5 lines
Quoted 1 line
And it fell back to no-inherit and succeeded.
On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)
Yep, your kernel is recent and has the following commit.
7e8b255650fcfa1d0("perf: SupportPERF_SAMPLE_READwith inherit")My kernel is 6.6 and it rejects such a combination. I'll test it on newer kernels later.
Quoted 43 lines
It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.
if (task && group_leader && group_leader->attr.inherit != attr.inherit) { err = -EINVAL; goto err_task; }Quoted 23 lines
I've also found that it
worked even withprecise_ip= 3.$ perf record -e cpu/mem-loads-aux/PS -vv true |& less ... sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 precise_ip 3 <<<---- hereQuoted 8 lines
sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 ------------------------------------------------------------ sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 = 4 ...Quoted 3 lines
Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test,
glc_get_event_constraints()successfully forces this event (config ==0x8203) to fixed counter 0, so there’s no issue here.Quoted 7 lines
That means your missing_features.inherit_sample_read should not be set. It's strange you have that with the recent kernels.
Can you run these commands and show the output here?
$ perf record -e task-clock:S true $ perf evlist -vOn 6.18.0-rc4:
$ perf record -e task-clock:S true [ perf record: Woken up 2 times to write data ] [ perf record: Captured and wrote 0.006 MB perf.data ]
Quoted 7 lines
$ perf evlist -v
task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1
(PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000,
sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled:
1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq:
1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1,
ksymbol: 1, bpf_event: 1, build_id: 1Thanks for sharing this. Yep, it has the inherit bit.
I think there's a bug in the missing feature test. Indeed, it should also have PERF_SAMPLE_TID for the test according to the kernel comment.
/* * We do not support PERF_SAMPLE_READ on inherited events unless * PERF_SAMPLE_TID is also selected, which allows inherited events to * collect per-thread samples. * See perf_output_read(). */ if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPLE_TID)) return ERR_PTR(-EINVAL);
I'll send a patch soon.
Thanks,
Namhyung
Quoted 194 lines · Namhyung Kim
Quoted 180 lines · Zide
Quoted 163 lines · Namhyung Kim
Quoted 102 lines · Zide
Quoted 99 lines · Namhyung Kim
Quoted 59 lines · Zide
Quoted 56 lines · Namhyung Kim
Hello,
Sorry for the delay.
Quoted 50 lines · Zide
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Zide
Quoted 13 lines · Namhyung Kim
Hello,
Quoted 10 lines · Zide Chen
Commit
c33aea446bf555ab("perf tools: Fixprecise_ipfallback logic") unconditionally called theprecise_ipfallback and moved it after the missing-feature checks so that it could handle EINVAL as well.However, this introduced an issue: after disabling missing features, the event could fail to open, which makes the subsequent
precise_ipfallback useless since it will always fail.For example, run the following command on Intel SPR:
Quoted 8 lines
Quoted 4 lines
$ perf record -e '{cpu/mem-loads-aux/S,cpu/mem-loads,ldlat=3/PS}' -- lsOpening the event "cpu/mem-loads,ldlat=3/PS" returns EINVAL when
precise_ip== 3. It then sets attr.inherit = false, which triggers aI'm curious about this part. Why the kernel set 'inherit = false'? IOW how did the leader event (mem-loads-aux) succeed with inherit = true then?
Initially, the inherit = true for both the group leader (cpu/mem-loads-aux/S) and the event in question (cpu/mem-loads,ldlat=3/PS).
Quoted 7 lines
Quoted 4 lines
When the second event fails with EINVAL, the current logic calls
evsel__detect_missing_features()first. Since this is aPERF_SAMPLE_READevent, the inherit attribute falls back to false, according to the fallback order implemented inevsel__detect_missing_features().Right, that means the kernel doesn't support
PERF_SAMPLE_READwith inherit = true. How did the first event succeed to open then?The perf tool sets
PERF_SAMPLE_TIDfor Inherit +PERF_SAMPLE_READevents, as implemented in commit90035d3cd876("tools/perf: Allow inherit +PERF_SAMPLE_READwhen opening event").Meanwhile, commit
7e8b255650fc("perf: SupportPERF_SAMPLE_READwith inherit") rejects a perf event ifhas_inherit_and_sample_read(attr)is true andPERF_SAMPLE_TIDis not set inattr->sample_type.Therefore, the first event succeeded, while the one opened in
evsel__detect_missing_features()which doesn't havePERF_SAMPLE_TIDfailed.Quoted 6 lines
Quoted 2 lines
Why does the first succeed and the second fail? Don't they have the same
SAMPLE_READandSAMPLE_TID+ inherit flags?Sorry, my previous reply wasn’t entirely accurate. The first event (cpu/mem-loads-aux/S) succeeds because it’s not a precise event (
precise_ip== 0).I'm not sure how it matters. I've tested the same command line on SPR and got this message. It says it failed to open because of inherit and
SAMPE_READ. It didn't haveprecise_iptoo.$ perf record -e cpu/mem-loads-aux/S -vv true |& less ... ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 inherit 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 ------------------------------------------------------------ sys_perf_event_open: pid 1161023 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ...Quoted 5 lines
Quoted 1 line
And it fell back to no-inherit and succeeded.
On my SPR, with either kernel 6.18.0-rc4 or the older 6.17.0-rc6, my test results are different from yours — I didn’t see any EINVAL, and there was no fallback. :)
Yep, your kernel is recent and has the following commit.
7e8b255650fcfa1d0("perf: SupportPERF_SAMPLE_READwith inherit")My kernel is 6.6 and it rejects such a combination. I'll test it on newer kernels later.
Quoted 43 lines
It’s strange, but even so, since there’s no group leader in this case, I assume that when it falls back to non-inherit, it should pass the following check.
if (task && group_leader && group_leader->attr.inherit != attr.inherit) { err = -EINVAL; goto err_task; }Quoted 23 lines
I've also found that it
worked even withprecise_ip= 3.$ perf record -e cpu/mem-loads-aux/PS -vv true |& less ... sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 sys_perf_event_open failed, error -22 Using PERF_SAMPLE_READ / :S modifier is not compatible with inherit, falling back to no-inherit. ------------------------------------------------------------ perf_event_attr: type 4 (cpu) size 136 config 0x8203 (mem-loads-aux) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|ID|PERIOD read_format ID|LOST disabled 1 mmap 1 comm 1 freq 1 enable_on_exec 1 task 1 precise_ip 3 <<<---- hereQuoted 8 lines
sample_id_all 1 mmap2 1 comm_exec 1 ksymbol 1 bpf_event 1 ------------------------------------------------------------ sys_perf_event_open: pid 1172834 cpu 0 group_fd -1 flags 0x8 = 4 ...Quoted 3 lines
Again, on my machine, I didn’t see EINVAL, and no fallback to non-inherit. In my test,
glc_get_event_constraints()successfully forces this event (config ==0x8203) to fixed counter 0, so there’s no issue here.Quoted 7 lines
That means your missing_features.inherit_sample_read should not be set. It's strange you have that with the recent kernels.
Can you run these commands and show the output here?
$ perf record -e task-clock:S true $ perf evlist -vOn 6.18.0-rc4:
$ perf record -e task-clock:S true [ perf record: Woken up 2 times to write data ] [ perf record: Captured and wrote 0.006 MB perf.data ]Quoted 7 lines
$ perf evlist -v task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1 (PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled: 1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq: 1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1, build_id: 1Thanks for sharing this. Yep, it has the inherit bit.
I think there's a bug in the missing feature test. Indeed, it should also have
PERF_SAMPLE_TIDfor the test according to the kernel comment.
Quoted 8 lines
/* * We do not support PERF_SAMPLE_READ on inherited events unless * PERF_SAMPLE_TID is also selected, which allows inherited events to * collect per-thread samples. * See perf_output_read(). */ if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPLE_TID)) return ERR_PTR(-EINVAL);
It seems that the purpose of the inherit_sample_read fallback is to remove the inherit attribute when both PERF_SAMPLE_READ and inherit are present, but PERF_SAMPLE_TID is not. The new change may not be able to accomplish this?
Quoted 6 lines
I'll send a patch soon.
Thanks,
Namhyung
Namhyung Kim
maintainerQuoted 45 lines · Zide
Quoted 33 lines · Namhyung Kim
Quoted 19 lines · Zide
Quoted 9 lines · Namhyung Kim
Quoted 3 lines · Zide
On 11/6/2025 10:52 AM, Namhyung Kim wrote:
Can you run these commands and show the output here?
$ perf record -e task-clock:S true $ perf evlist -vOn 6.18.0-rc4:
$ perf record -e task-clock:S true [ perf record: Woken up 2 times to write data ] [ perf record: Captured and wrote 0.006 MB perf.data ]Quoted 7 lines
$ perf evlist -v task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1 (PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled: 1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq: 1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1, build_id: 1Thanks for sharing this. Yep, it has the inherit bit.
I think there's a bug in the missing feature test. Indeed, it should also have
PERF_SAMPLE_TIDfor the test according to the kernel comment.Quoted 8 lines
/* * We do not support PERF_SAMPLE_READ on inherited events unless * PERF_SAMPLE_TID is also selected, which allows inherited events to * collect per-thread samples. * See perf_output_read(). */ if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPLE_TID)) return ERR_PTR(-EINVAL);
Quoted 4 lines
It seems that the purpose of the
inherit_sample_readfallback is to remove the inherit attribute when bothPERF_SAMPLE_READand inherit are present, butPERF_SAMPLE_TIDis not. The new change may not be able to accomplish this?
No, the purpose of the missing feature check is to detect whether the current kernel supports this feature or not. The correct check should pass both READ and TID together.
Thanks,
Namhyung
Quoted 46 lines · Namhyung Kim
Quoted 45 lines · Zide
Quoted 41 lines · Namhyung Kim
Quoted 19 lines · Zide
Quoted 9 lines · Namhyung Kim
Quoted 3 lines · Zide
On 11/6/2025 10:52 AM, Namhyung Kim wrote:
Can you run these commands and show the output here?
$ perf record -e task-clock:S true $ perf evlist -vOn 6.18.0-rc4:
$ perf record -e task-clock:S true [ perf record: Woken up 2 times to write data ] [ perf record: Captured and wrote 0.006 MB perf.data ]Quoted 7 lines
$ perf evlist -v task-clock:Su: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x1 (PERF_COUNT_SW_TASK_CLOCK), { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|READ|ID|PERIOD, read_format: ID|LOST, disabled: 1, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, freq: 1, enable_on_exec: 1, task: 1, sample_id_all: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1, build_id: 1Thanks for sharing this. Yep, it has the inherit bit.
I think there's a bug in the missing feature test. Indeed, it should also have
PERF_SAMPLE_TIDfor the test according to the kernel comment./* * We do not support PERF_SAMPLE_READ on inherited events unless * PERF_SAMPLE_TID is also selected, which allows inherited events to * collect per-thread samples. * See perf_output_read(). */ if (has_inherit_and_sample_read(attr) && !(attr->sample_type & PERF_SAMPLE_TID)) return ERR_PTR(-EINVAL);
Quoted 8 lines
Quoted 4 lines
It seems that the purpose of the
inherit_sample_readfallback is to remove the inherit attribute when bothPERF_SAMPLE_READand inherit are present, butPERF_SAMPLE_TIDis not. The new change may not be able to accomplish this?No, the purpose of the missing feature check is to detect whether the current kernel supports this feature or not. The correct check should pass both READ and TID together.
OK, thanks!
Quoted 3 lines
Thanks,
Namhyung