perf synthetic-events: Fix stale build ID in module MMAP2 records
PATCH 1/1 perf synthetic-events: Fix stale build ID in module MMAP2 records
From: Chuck Lever <[email protected]>
perf_event__synthesize_modules() allocates a single union perf_event and reuses it across every kernel module callback. After the first module is processed, perf_record_mmap2__read_build_id() sets PERF_RECORD_MISC_MMAP_BUILD_ID in header.misc and writes that module's build ID into the event. On subsequent iterations the callback overwrites start, len, pid, and filename for the next module but never clears the stale build ID fields or the MMAP_BUILD_ID flag. When perf_record_mmap2__read_build_id() runs for the second module it sees the flag, reads the stale build ID into a dso_id, and __dso__improve_id() permanently poisons the DSO with the wrong build ID. Every module after the first therefore receives the first module's build ID in its MMAP2 record. On a system with the sunrpc and nfsd modules loaded, this causes perf script and perf report to show [unknown] for all module symbols.
The latent bug has existed since commit d9f2ecbc5e47 ("perf
dso: Move build_id to dso_id") introduced the PERF_RECORD_MISC_MMAP_BUILD_ID check in perf_record_mmap2__read_build_id(). Commit 53b00ff358dc ("perf
record: Make --buildid-mmap the default") then exposed it to all users by making the MMAP2-with-build-ID path the default. Both commits were merged in the same series.
Clear the MMAP_BUILD_ID flag and zero the build_id union before each call to perf_record_mmap2__read_build_id() so that every module starts with a clean slate.
--- tools/perf/util/synthetic-events.c | 5 +++++ 1 file changed, 5 insertions(+)
tools/perf/util/synthetic-events.c +5 -0
@@ -703,6 +703,11 @@ static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data)memcpy (event->mmap2.filename,dso__long_name (dso),dso__long_name_len (dso) +1 ); +/* Clear stale build ID from previous module iteration */ + event->mmap2.header.misc &= ~PERF_RECORD_MISC_MMAP_BUILD_ID ; +memset (event->mmap2.build_id,0 ,sizeof (event->mmap2.build_id)); + event->mmap2.build_id_size =0 ; +perf_record_mmap2__read_build_id (&event->mmap2, args->machine, false); }else { size =PERF_ALIGN (dso__long_name_len (dso) +1 ,sizeof (u64));
Signature · 1 line
2.53.0
Quoted 43 lines · Chuck Lever
From: Chuck Lever <[email protected]>
perf_event__synthesize_modules()allocates a single unionperf_eventand reuses it across every kernel module callback. After the first module is processed,perf_record_mmap2__read_build_id()setsPERF_RECORD_MISC_MMAP_BUILD_IDin header.misc and writes that module's build ID into the event. On subsequent iterations the callback overwrites start, len, pid, and filename for the next module but never clears the stale build ID fields or theMMAP_BUILD_IDflag. Whenperf_record_mmap2__read_build_id()runs for the second module it sees the flag, reads the stale build ID into adso_id, and__dso__improve_id()permanently poisons the DSO with the wrong build ID. Every module after the first therefore receives the first module's build ID in its MMAP2 record. On a system with the sunrpc and nfsd modules loaded, this causes perf script and perf report to show [unknown] for all module symbols.The latent bug has existed since commit
d9f2ecbc5e47("perf
dso: Movebuild_idtodso_id") introduced thePERF_RECORD_MISC_MMAP_BUILD_IDcheck inperf_record_mmap2__read_build_id(). Commit53b00ff358dc("perf
record: Make --buildid-mmap the default") then exposed it to all users by making the MMAP2-with-build-ID path the default. Both commits were merged in the same series.Clear the
MMAP_BUILD_IDflag and zero thebuild_idunion before each call toperf_record_mmap2__read_build_id()so that every module starts with a clean slate.Fixes d9f2ecbc5e47 ("perf dso: Move build_id to dso_id")Signed-off-by Chuck Lever <[email protected]>--- tools/perf/util/synthetic-events.c | 5 +++++ 1 file changed, 5 insertions(+)tools/perf/util/synthetic-events.c +0
-0
tools/perf/util/synthetic-events.c static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data) · line 711
Quoted 9 lines
tools/perf/util/synthetic-events.c +5
-0@@ -703,6 +703,11 @@ static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data)memcpy (event->mmap2.filename,dso__long_name (dso),dso__long_name_len (dso) +1 ); +/* Clear stale build ID from previous module iteration */ + event->mmap2.header.misc &= ~PERF_RECORD_MISC_MMAP_BUILD_ID ; +memset (event->mmap2.build_id,0 ,sizeof (event->mmap2.build_id)); + event->mmap2.build_id_size =0 ; +perf_record_mmap2__read_build_id (&event->mmap2, args->machine, false);
Thanks for the report and fix. One thing this change will fix is the dso lookup, as the previous mmap2 event's build id would be read by perf_record_mmap2__read_build_id for the dso_id. The other fix addresses the written out mmap2 event, which now won't contain data from the last event when reading a module's build ID fails. If reading the build ID fails, since the build id misc bit is clear we should probably write the maj, min, ino and ino_generation fields. In perf_event__synthesize_mmap_events these are populated by the reading of the /proc/pid/maps file but in perf_event__synthesize_modules_maps_cb and __perf_event__synthesize_kernel_mmap we probably need to stat the file if possible - mind you the non-build ID code there uses an mmap event rather than an mmap2 event, where these fields aren't present.
So:
But I think we can improve the perf_record_mmap2__read_build_id when filename__read_build_id fails.
Thanks,
Ian
Quoted 5 lines
tools/perf/util/synthetic-events.c +0
-1}else { size =PERF_ALIGN (dso__long_name_len (dso) +1 ,sizeof (u64)); --2.53.0
Ian Rogers
reviewerQuoted 40 lines · Ian Rogers
Quoted 38 lines · Chuck Lever
From: Chuck Lever <[email protected]>
perf_event__synthesize_modules()allocates a single unionperf_eventand reuses it across every kernel module callback. After the first module is processed,perf_record_mmap2__read_build_id()setsPERF_RECORD_MISC_MMAP_BUILD_IDin header.misc and writes that module's build ID into the event. On subsequent iterations the callback overwrites start, len, pid, and filename for the next module but never clears the stale build ID fields or theMMAP_BUILD_IDflag. Whenperf_record_mmap2__read_build_id()runs for the second module it sees the flag, reads the stale build ID into adso_id, and__dso__improve_id()permanently poisons the DSO with the wrong build ID. Every module after the first therefore receives the first module's build ID in its MMAP2 record. On a system with the sunrpc and nfsd modules loaded, this causes perf script and perf report to show [unknown] for all module symbols.The latent bug has existed since commit
d9f2ecbc5e47("perf
dso: Movebuild_idtodso_id") introduced thePERF_RECORD_MISC_MMAP_BUILD_IDcheck inperf_record_mmap2__read_build_id(). Commit53b00ff358dc("perf
record: Make --buildid-mmap the default") then exposed it to all users by making the MMAP2-with-build-ID path the default. Both commits were merged in the same series.Clear the
MMAP_BUILD_IDflag and zero thebuild_idunion before each call toperf_record_mmap2__read_build_id()so that every module starts with a clean slate.Fixes d9f2ecbc5e47 ("perf dso: Move build_id to dso_id")Signed-off-by Chuck Lever <[email protected]>--- tools/perf/util/synthetic-events.c | 5 +++++ 1 file changed, 5 insertions(+)
Quoted 7 lines
tools/perf/util/synthetic-events.c static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data) · line 704
Quoted 7 lines
tools/perf/util/synthetic-events.c +0
-0@@ -703,6 +703,11 @@ static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data)memcpy (event->mmap2.filename,dso__long_name (dso),dso__long_name_len (dso) +1 );
AI points out that leaving stale data isn't just a problem for misc bits but also after the filename, due to the event's alignment to 8 bytes. I think this is benign, you're only leaking bits of the filename from the last event, not uninitialized data from the perf execution. The AI made me think about it so I thought I'd repeat it. We could add a memset here to clear the filename up to the alignment, as is done here:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/synthetic-events.c?h=perf-tools-next#n529
```
memset(event->mmap2.filename + size, 0, machine->id_hdr_size +
(aligned_size - size));
```Thanks,
Ian
Quoted 35 lines
tools/perf/util/synthetic-events.c static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data) · line 711
Quoted 7 lines
tools/perf/util/synthetic-events.c +5
-0+/* Clear stale build ID from previous module iteration */ + event->mmap2.header.misc &= ~PERF_RECORD_MISC_MMAP_BUILD_ID ; +memset (event->mmap2.build_id,0 ,sizeof (event->mmap2.build_id)); + event->mmap2.build_id_size =0 ; +perf_record_mmap2__read_build_id (&event->mmap2, args->machine, false);Thanks for the report and fix. One thing this change will fix is the dso lookup, as the previous mmap2 event's build id would be read by
perf_record_mmap2__read_build_idfor thedso_id. The other fix addresses the written out mmap2 event, which now won't contain data from the last event when reading a module's build ID fails. If reading the build ID fails, since the build id misc bit is clear we should probably write the maj, min, ino andino_generationfields. Inperf_event__synthesize_mmap_eventsthese are populated by the reading of the/proc/pid/mapsfile but inperf_event__synthesize_modules_maps_cband__perf_event__synthesize_kernel_mmapwe probably need to stat the file if possible - mind you the non-build ID code there uses an mmap event rather than an mmap2 event, where these fields aren't present.So:
Reviewed-by Ian Rogers <[email protected]>But I think we can improve the
perf_record_mmap2__read_build_idwhenfilename__read_build_idfails.Thanks,
IanQuoted 5 lines
tools/perf/util/synthetic-events.c +0
-1}else { size =PERF_ALIGN (dso__long_name_len (dso) +1 ,sizeof (u64)); --2.53.0
Quoted 57 lines · Ian Rogers
Quoted 40 lines · Ian Rogers
Quoted 38 lines · Chuck Lever
From: Chuck Lever <[email protected]>
perf_event__synthesize_modules()allocates a single unionperf_eventand reuses it across every kernel module callback. After the first module is processed,perf_record_mmap2__read_build_id()setsPERF_RECORD_MISC_MMAP_BUILD_IDin header.misc and writes that module's build ID into the event. On subsequent iterations the callback overwrites start, len, pid, and filename for the next module but never clears the stale build ID fields or theMMAP_BUILD_IDflag. Whenperf_record_mmap2__read_build_id()runs for the second module it sees the flag, reads the stale build ID into adso_id, and__dso__improve_id()permanently poisons the DSO with the wrong build ID. Every module after the first therefore receives the first module's build ID in its MMAP2 record. On a system with the sunrpc and nfsd modules loaded, this causes perf script and perf report to show [unknown] for all module symbols.The latent bug has existed since commit
d9f2ecbc5e47("perf
dso: Movebuild_idtodso_id") introduced thePERF_RECORD_MISC_MMAP_BUILD_IDcheck inperf_record_mmap2__read_build_id(). Commit53b00ff358dc("perf
record: Make --buildid-mmap the default") then exposed it to all users by making the MMAP2-with-build-ID path the default. Both commits were merged in the same series.Clear the
MMAP_BUILD_IDflag and zero thebuild_idunion before each call toperf_record_mmap2__read_build_id()so that every module starts with a clean slate.Fixes d9f2ecbc5e47 ("perf dso: Move build_id to dso_id")Signed-off-by Chuck Lever <[email protected]>--- tools/perf/util/synthetic-events.c | 5 +++++ 1 file changed, 5 insertions(+)Quoted 7 lines
tools/perf/util/synthetic-events.c static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data) · line 704
Quoted 7 lines
tools/perf/util/synthetic-events.c +0
-0@@ -703,6 +703,11 @@ static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data)memcpy (event->mmap2.filename,dso__long_name (dso),dso__long_name_len (dso) +1 );AI points out that leaving stale data isn't just a problem for misc bits but also after the filename, due to the event's alignment to 8 bytes. I think this is benign, you're only leaking bits of the filename from the last event, not uninitialized data from the perf execution. The AI made me think about it so I thought I'd repeat it. We could add a memset here to clear the filename up to the alignment, as is done here:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/synthetic-events.c?h=perf-tools-next#n529
Quoted 4 lines
```
memset(event->mmap2.filename + size, 0, machine->id_hdr_size +
(aligned_size - size));
```Hi Ian,
I'm not attached to the exact way this issue is addressed, so if you know of improvements, please feel free to incorporate them.
Signature · 1 line
Chuck Lever
Quoted 68 lines · Ian Rogers
Quoted 43 lines · Chuck Lever
From: Chuck Lever <[email protected]>
perf_event__synthesize_modules()allocates a single unionperf_eventand reuses it across every kernel module callback. After the first module is processed,perf_record_mmap2__read_build_id()setsPERF_RECORD_MISC_MMAP_BUILD_IDin header.misc and writes that module's build ID into the event. On subsequent iterations the callback overwrites start, len, pid, and filename for the next module but never clears the stale build ID fields or theMMAP_BUILD_IDflag. Whenperf_record_mmap2__read_build_id()runs for the second module it sees the flag, reads the stale build ID into adso_id, and__dso__improve_id()permanently poisons the DSO with the wrong build ID. Every module after the first therefore receives the first module's build ID in its MMAP2 record. On a system with the sunrpc and nfsd modules loaded, this causes perf script and perf report to show [unknown] for all module symbols.The latent bug has existed since commit
d9f2ecbc5e47("perf
dso: Movebuild_idtodso_id") introduced thePERF_RECORD_MISC_MMAP_BUILD_IDcheck inperf_record_mmap2__read_build_id(). Commit53b00ff358dc("perf
record: Make --buildid-mmap the default") then exposed it to all users by making the MMAP2-with-build-ID path the default. Both commits were merged in the same series.Clear the
MMAP_BUILD_IDflag and zero thebuild_idunion before each call toperf_record_mmap2__read_build_id()so that every module starts with a clean slate.Fixes d9f2ecbc5e47 ("perf dso: Move build_id to dso_id")Signed-off-by Chuck Lever <[email protected]>--- tools/perf/util/synthetic-events.c | 5 +++++ 1 file changed, 5 insertions(+)tools/perf/util/synthetic-events.c +0
-0tools/perf/util/synthetic-events.c static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data) · line 711
Quoted 9 lines
tools/perf/util/synthetic-events.c +5
-0@@ -703,6 +703,11 @@ static int perf_event__synthesize_modules_maps_cb(struct map *map, void *data)memcpy (event->mmap2.filename,dso__long_name (dso),dso__long_name_len (dso) +1 ); +/* Clear stale build ID from previous module iteration */ + event->mmap2.header.misc &= ~PERF_RECORD_MISC_MMAP_BUILD_ID ; +memset (event->mmap2.build_id,0 ,sizeof (event->mmap2.build_id)); + event->mmap2.build_id_size =0 ; +perf_record_mmap2__read_build_id (&event->mmap2, args->machine, false);Thanks for the report and fix. One thing this change will fix is the dso lookup, as the previous mmap2 event's build id would be read by
perf_record_mmap2__read_build_idfor thedso_id. The other fix addresses the written out mmap2 event, which now won't contain data from the last event when reading a module's build ID fails. If reading the build ID fails, since the build id misc bit is clear we should probably write the maj, min, ino andino_generationfields. Inperf_event__synthesize_mmap_eventsthese are populated by the reading of the/proc/pid/mapsfile but inperf_event__synthesize_modules_maps_cband__perf_event__synthesize_kernel_mmapwe probably need to stat the file if possible - mind you the non-build ID code there uses an mmap event rather than an mmap2 event, where these fields aren't present.
Quoted 4 lines
So:
Reviewed-by Ian Rogers <[email protected]>But I think we can improve the
perf_record_mmap2__read_build_idwhenfilename__read_build_idfails.
Sure, but lets get this reviewed fix in for now, thanks to you both,
Thanks, applied to perf-tools, for v7.0.
- Arnaldo
Quoted 8 lines
Thanks,
IanQuoted 5 lines
tools/perf/util/synthetic-events.c +0
-1}else { size =PERF_ALIGN (dso__long_name_len (dso) +1 ,sizeof (u64)); --2.53.0