python3-perf pulling dependencies after build refactoring
Hello Ian,
Jitka Plesnikova has spotted an interesting fact about he python bindings -- perf.cpython-*-x86_64-linux-gnu.so library -- the number of shared objects it links against has significantly grown between v6.10 and v6.11:
Before:
# ldd python/perf.cpython-39-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffefd1cb000)
libm.so.6 => /lib64/libm.so.6 (0x00007f4c9315e000)
libz.so.1 => /lib64/libz.so.1 (0x00007f4c93144000)
libelf.so.1 => /lib64/libelf.so.1 (0x00007f4c929e4000)
libslang.so.2 => /lib64/libslang.so.2 (0x00007f4c92600000)
libc.so.6 => /lib64/libc.so.6 (0x00007f4c92200000)
libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f4c93136000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4c93247000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f4c9290d000)After:
# ldd python/perf.cpython-39-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffd889f3000)
libm.so.6 => /lib64/libm.so.6 (0x00007f75cd125000)
libz.so.1 => /lib64/libz.so.1 (0x00007f75cdbc0000)
libelf.so.1 => /lib64/libelf.so.1 (0x00007f75cdba4000)
libdw.so.1 => /lib64/libdw.so.1 (0x00007f75cd089000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f75ccc00000)
libslang.so.2 => /lib64/libslang.so.2 (0x00007f75cc800000)
libperl.so.5.32 => /lib64/libperl.so.5.32 (0x00007f75cc400000)
libc.so.6 => /lib64/libc.so.6 (0x00007f75cc000000)
libpython3.9.so.1.0 => /lib64/libpython3.9.so.1.0 (0x00007f75cbc00000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f75cb800000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f75cdb76000)
libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f75ccb29000)
libcap.so.2 => /lib64/libcap.so.2 (0x00007f75cdb6a000)
libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f75cdb5c000)
libbabeltrace-ctf.so.1 => /lib64/libbabeltrace-ctf.so.1 (0x00007f75cd03f000)
libpfm.so.4 => /lib64/libpfm.so.4 (0x00007f75cb400000)
libtraceevent.so.1 => /lib64/libtraceevent.so.1 (0x00007f75ccb08000)
/lib64/ld-linux-x86-64.so.2 (0x00007f75cdbe8000)
libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f75cdb49000)
libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f75cc7c6000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f75ccaed000)
libbabeltrace.so.1 => /lib64/libbabeltrace.so.1 (0x00007f75ccadd000)
libpopt.so.0 => /lib64/libpopt.so.0 (0x00007f75cc7b7000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f75cdb3d000)
libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f75cd039000)
libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f75cc2c5000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f75cc24d000)I have narrowed it down to e467705a9fb3 ("perf util: Make util its own library"). It has also grown in size, probably because of this linking change. I think we need to get rid of this probably unwanted side-effect of the build optimization. What do you think?
Regards,
Michael
Quoted 52 lines · Michael Petlan
Hello Ian,
Jitka Plesnikova has spotted an interesting fact about he python bindings -- perf.cpython-*-x86_64-linux-gnu.so library -- the number of shared objects it links against has significantly grown between v6.10 and v6.11:
Before:
# ldd python/perf.cpython-39-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007ffefd1cb000) libm.so.6 => /lib64/libm.so.6 (0x00007f4c9315e000) libz.so.1 => /lib64/libz.so.1 (0x00007f4c93144000) libelf.so.1 => /lib64/libelf.so.1 (0x00007f4c929e4000) libslang.so.2 => /lib64/libslang.so.2 (0x00007f4c92600000) libc.so.6 => /lib64/libc.so.6 (0x00007f4c92200000) libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f4c93136000) /lib64/ld-linux-x86-64.so.2 (0x00007f4c93247000) libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f4c9290d000)After:
# ldd python/perf.cpython-39-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007ffd889f3000) libm.so.6 => /lib64/libm.so.6 (0x00007f75cd125000) libz.so.1 => /lib64/libz.so.1 (0x00007f75cdbc0000) libelf.so.1 => /lib64/libelf.so.1 (0x00007f75cdba4000) libdw.so.1 => /lib64/libdw.so.1 (0x00007f75cd089000) libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f75ccc00000) libslang.so.2 => /lib64/libslang.so.2 (0x00007f75cc800000) libperl.so.5.32 => /lib64/libperl.so.5.32 (0x00007f75cc400000) libc.so.6 => /lib64/libc.so.6 (0x00007f75cc000000) libpython3.9.so.1.0 => /lib64/libpython3.9.so.1.0 (0x00007f75cbc00000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f75cb800000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f75cdb76000) libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f75ccb29000) libcap.so.2 => /lib64/libcap.so.2 (0x00007f75cdb6a000) libnuma.so.1 => /lib64/libnuma.so.1 (0x00007f75cdb5c000) libbabeltrace-ctf.so.1 => /lib64/libbabeltrace-ctf.so.1 (0x00007f75cd03f000) libpfm.so.4 => /lib64/libpfm.so.4 (0x00007f75cb400000) libtraceevent.so.1 => /lib64/libtraceevent.so.1 (0x00007f75ccb08000) /lib64/ld-linux-x86-64.so.2 (0x00007f75cdbe8000) libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f75cdb49000) libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f75cc7c6000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f75ccaed000) libbabeltrace.so.1 => /lib64/libbabeltrace.so.1 (0x00007f75ccadd000) libpopt.so.0 => /lib64/libpopt.so.0 (0x00007f75cc7b7000) libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f75cdb3d000) libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f75cd039000) libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f75cc2c5000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f75cc24d000)
Quoted 4 lines
I have narrowed it down to
e467705a9fb3("perf util: Make util its own library"). It has also grown in size, probably because of this linking change. I think we need to get rid of this probably unwanted side-effect of the build optimization. What do you think?
Hi Michael,
it wasn't a build optimization. The problem was that we have things like lex and yacc generated C code that things like parse_events require to, you know, be able to parse events :-) So that we don't duplicate logic from event parsing for things like heterogeneous/hybrid/BIG.little machines we've been moving logic in places like evlist to use parse_events rather than hand craft struct perf_event_attr. So evlist has a dependence on parse_events but previously the python code would stub out parse_events meaning the python use of evlist had regressed.
I tried a few times to restructure things so that python and the build would work as it was, but in the end the simplest solution was to break apart the build. That is make util, tests, ui, bench.. their own library and then link these with the builtin and perf.c code to make the executable. Then the python code could depend on these intermediate libraries, not worry about how they were made (lex/yacc building, etc.), and we could have the full parse_events code in the python code.
Now the code in util has additional dependencies, and we're linking against libraries with those dependencies, so that's why the size has gone up. Can we do better? Yes, we can. In these two patches we remove the test and bench libraries from the python build:
https://lore.kernel.org/lkml/[email protected]/
(unmerged)
https://lore.kernel.org/lkml/[email protected]/
(unmerged)
The removal of the bench library will remove the use of libnuma.
Can we do yet better? Yes, if we can decouple the util and ui code then there is no reason libslang be part of the python code. Unfortunately that code is quite interwoven currently.
Can we do even more better? Possibly, in these changes libtraceevent becomes more optional:
https://lore.kernel.org/lkml/[email protected]/
(unmerged)
But removing this would also remove libtracevent support from perf, so this probably isn't something you'd want to enable.
The libcap dependency should be gone after:
https://lore.kernel.org/lkml/[email protected]/
libpfm is tied to event parsing so we probably want to keep it as a dependence.
libbabeltrace I thought was used for syscall tables when other things fail. Hopefully we can just have syscall tables and do away with this. This cleanup makes me excited:
https://lore.kernel.org/linux-perf-users/Zyk9hX8CB_2rbWsi@ghost/T/#t
The compression library stuff is so that compressed events can work, hard to see how to remove these without disabling compression support.
As perf script can run python and perl code we get those dependencies, maybe we can get the relevant code only to be part of the builtin-script.c and lose those dependencies from the util code.
libglib and libmodule I think relate to the not very maintained GTk code. We probably should delete that code from the build. If we decouple util and ui then again I think the dependency goes.
libpopt, I don't know where this dependency is coming from but it looks bogus. perf's command line processing comes from the statically linked libsubcommand.
libcrypt and lubuuid I think are only needed for genelf which is part of perf inject -j. Possibly this logic can be carved out of util to avoid bringing in the dependencies.
Thanks,
Ian