Build trace-cruncher as Python pakage
v2 cover Build trace-cruncher as Python pakage
This patch-set is an attempt to restructure the project and to make it build as a native Python package. Although it looks like a complete rewrite, this is essentially just a switching from using Cython to using directly the C API of Python. Cython is still being used but only for the implementation of the NumPy data wrapper. The new building procedure is very simple and intuitive. The appropriate versions of trace-cmd and KernelShark libraries are automatically download, patch and build. These third-party libraries are installed as part of trace-cruncher itself and do not interfere with any existing system-wide installations of trace-cmd and KernelShark.
All patches are available at:
https://github.com/vmware/trace-cruncher/tree/refactoring_WIP (branch "refactoring_WIP")
v2 changes:
- Trying to address the building issues reported by Douglas Raillard in particular the missing header file and the problem of linking with libkshark.so.X.Y.Z
- Douglas suggested to use venv in order to avoid the patched versions of the third-party libraries to pollut the system-wide locations. Here I am suggesting a different solution that allows to make those third-party libraries part of the package itself. Not sure which of the two solution is the best. It will be great if we can have a discussion.
- All obsoleted source files are removed and the README file is updated to describe the new building procedure.
Yordan Karadzhov (VMware) (12):
trace-cruncher: Refactor the part of the interface that relies on
libkshark
trace-cruncher: Refactor the part of the interface that relies on
libtraceevent
trace-cruncher: Refactor NumPy based data wrapper
trace-cruncher: Add "utils"
trace-cruncher: Adapt sched_wakeup.py to use the new module
trace-cruncher: Add Makefile
trace-cruncher: Adapt gpareto_fit.py to use the new module
trace-cruncher: Adapt page_faults.py to use the new module
trace-cruncher: Automate the third-party build
Update README.md
trace-cruncher: Remove all leftover files.
trace-cruncher: Improve Makefile Provide more robust and better
looking build process.
...-_DEVEL-_LIBS-and-_RPATH_TO_ORIGIN-b.patch | 160 ++++++++ 0001-kernel-shark-Add-_DEVEL-build-flag.patch | 90 ----- 0002-kernel-shark-Add-reg_pid-plugin.patch | 8 +- Makefile | 36 ++ README.md | 50 +-- clean.sh | 6 - examples/gpareto_fit.py | 35 +- examples/ksharksetup.py | 24 -- examples/page_faults.py | 64 ++-- examples/sched_wakeup.py | 30 +- install_third_party.sh | 39 ++ libkshark-py.c | 224 ----------- libkshark_wrapper.pyx | 361 ------------------ np_setup.py | 90 ----- setup.py | 68 ++++ src/common.h | 20 + src/datawrapper.pyx | 201 ++++++++++ src/ftracepy.c | 233 +++++++++++ src/ksharkpy.c | 269 +++++++++++++ src/trace2matrix.c | 29 ++ tracecruncher/__init__.py | 4 + tracecruncher/utils.py | 54 +++ 22 files changed, 1188 insertions(+), 907 deletions(-) create mode 100644 0001-kernel-shark-Add-_DEVEL-_LIBS-and-_RPATH_TO_ORIGIN-b.patch delete mode 100644 0001-kernel-shark-Add-_DEVEL-build-flag.patch create mode 100644 Makefile delete mode 100755 clean.sh delete mode 100644 examples/ksharksetup.py create mode 100755 install_third_party.sh delete mode 100644 libkshark-py.c delete mode 100644 libkshark_wrapper.pyx delete mode 100755 np_setup.py create mode 100644 setup.py create mode 100644 src/common.h create mode 100644 src/datawrapper.pyx create mode 100644 src/ftracepy.c create mode 100644 src/ksharkpy.c create mode 100644 src/trace2matrix.c create mode 100644 tracecruncher/__init__.py create mode 100644 tracecruncher/utils.py
Signature · 1 line
2.20.1
PATCH v2 1/12 trace-cruncher: Refactor the part of the interface that relies on libkshark
This is the first patch from a patch-set that aims to refactor trace-cruncher completely. The goal it to be able to build the project as a native Python package, which contains several sub-packages implemented as C extensions via the Python's C API. In this patch the part of the interface that relies on libkshark gets re-implemented as an extension called "tracecruncher.ksharkpy". Note that this new extension has a stand-alone build that is completely decoupled from the existing build system used by trace-cruncher.
--- setup.py | 43 ++++++ src/common.h | 19 +++ src/ksharkpy.c | 267 ++++++++++++++++++++++++++++++++++++++ tracecruncher/__init__.py | 0 4 files changed, 329 insertions(+) create mode 100644 setup.py create mode 100644 src/common.h create mode 100644 src/ksharkpy.c create mode 100644 tracecruncher/__init__.py
setup.py new file +43 -0
@@ -0,0 +1,43 @@ +#!/usr/bin/env python3 + +""" +SPDX-License-Identifier: LGPL-2.1 + +Copyright 2019 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> +""" + +from setuptools import setup, find_packages +from distutils.core import Extension +from Cython.Build import cythonize + +def main(): + kshark_path = '/usr/local/lib/kernelshark' + + module_ks = Extension('tracecruncher.ksharkpy', + sources=['src/ksharkpy.c'], + library_dirs=[kshark_path], + runtime_library_dirs=[kshark_path], + libraries=['kshark'], + define_macros=[ + ('LIB_KSHARK_PATH', '\"' + kshark_path + '/libkshark.so\"'), + ('KS_PLUGIN_DIR', '\"' + kshark_path + '/plugins\"') + ], + ) + + setup(name='tracecruncher', + version='0.1.0', + description='NumPy based interface for accessing tracing data in Python.', + author='Yordan Karadzhov (VMware)', + author_email='[email protected]', + url='https://github.com/vmware/trace-cruncher', + license='LGPL-2.1', + packages=find_packages(), + ext_modules=[module_ks], + classifiers=[ + 'Development Status :: 3 - Alpha', + 'Programming Language :: Python :: 3', + ] + ) + +if __name__ == '__main__': + main()
src/common.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: LGPL-2.1 */ + +/* +* Copyright (C) 2017 VMware Inc, Yordan Karadzhov <[email protected]> +*/ + +#ifndef _TC_COMMON_H +#define _TC_COMMON_H + +#define TRACECRUNCHER_ERROR tracecruncher_error +#define KSHARK_ERROR kshark_error + +#define KS_INIT_ERROR \ + PyErr_SetString(KSHARK_ERROR, "libshark failed to initialize"); + +#define KS_MEM_ERROR \ + PyErr_SetString(TRACECRUNCHER_ERROR, "failed to allocate memory"); + +#endif
src/ksharkpy.c new file +267 -0
267 lines not on this page · raw message
tracecruncher/__init__.py new file +0 -0
Signature · 1 line
2.20.1
PATCH v2 2/12 trace-cruncher: Refactor the part of the interface that relies on libtraceevent
In reply to Yordan Karadzhov (VMware)The part of the interface that relies on libtraceevent gets re-implemented as an extension called "tracecruncher.ftracepy". The new extension gets build together with the previously implemented "tracecruncher.ksharkpy" extension.
--- setup.py | 11 ++- src/common.h | 1 + src/ftracepy.c | 233 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 244 insertions(+), 1 deletion(-) create mode 100644 src/ftracepy.c
setup.py +10 -1
@@ -12,6 +12,8 @@ from Cython.Build import cythonize def main(): kshark_path = '/usr/local/lib/kernelshark' + traceevent_path = '/usr/local/lib/traceevent/' + tracecmd_path = '/usr/local/lib/trace-cmd/' module_ks = Extension('tracecruncher.ksharkpy', sources=['src/ksharkpy.c'],
@@ -24,6 +26,13 @@ def main(): ], ) + module_ft = Extension('tracecruncher.ftracepy', + sources=['src/ftracepy.c'], + library_dirs=[kshark_path, traceevent_path, tracecmd_path], + runtime_library_dirs=[kshark_path, traceevent_path, tracecmd_path], + libraries=['kshark', 'traceevent', 'tracecmd'], + ) + setup(name='tracecruncher', version='0.1.0', description='NumPy based interface for accessing tracing data in Python.',
@@ -32,7 +41,7 @@ def main(): url='https://github.com/vmware/trace-cruncher', license='LGPL-2.1', packages=find_packages(), - ext_modules=[module_ks], + ext_modules=[module_ks, module_ft], classifiers=[ 'Development Status :: 3 - Alpha', 'Programming Language :: Python :: 3',
src/common.h +1 -0
@@ -9,6 +9,7 @@#define TRACECRUNCHER_ERROR tracecruncher_error#define KSHARK_ERROR kshark_error +#define FTRACE_ERROR ftrace_error#define KS_INIT_ERROR \ PyErr_SetString(KSHARK_ERROR, "libshark failed to initialize");
src/ftracepy.c new file +233 -0
233 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v2 3/12 trace-cruncher: Refactor NumPy based data wrapper
In reply to Yordan Karadzhov (VMware)The data wrapper is the only thing that remains being built with Cython. It is now a subpackage called "tracecruncher.datawrapper".
--- setup.py | 12 ++- src/datawrapper.pyx | 201 ++++++++++++++++++++++++++++++++++++++++++++ src/trace2matrix.c | 29 +++++++ 3 files changed, 241 insertions(+), 1 deletion(-) create mode 100644 src/datawrapper.pyx create mode 100644 src/trace2matrix.c
setup.py +11 -1
@@ -9,12 +9,22 @@ Copyright 2019 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> from setuptools import setup, find_packages from distutils.core import Extension from Cython.Build import cythonize +import numpy as np def main(): kshark_path = '/usr/local/lib/kernelshark' traceevent_path = '/usr/local/lib/traceevent/' tracecmd_path = '/usr/local/lib/trace-cmd/' + cythonize('src/datawrapper.pyx') + module_data = Extension('tracecruncher.datawrapper', + sources=['src/datawrapper.c'], + include_dirs=[np.get_include()], + library_dirs=[kshark_path, traceevent_path, tracecmd_path], + runtime_library_dirs=[kshark_path, traceevent_path, tracecmd_path], + libraries=['kshark', 'traceevent', 'tracecmd'] + ) + module_ks = Extension('tracecruncher.ksharkpy', sources=['src/ksharkpy.c'], library_dirs=[kshark_path],
@@ -41,7 +51,7 @@ def main(): url='https://github.com/vmware/trace-cruncher', license='LGPL-2.1', packages=find_packages(), - ext_modules=[module_ks, module_ft], + ext_modules=[module_data, module_ks, module_ft], classifiers=[ 'Development Status :: 3 - Alpha', 'Programming Language :: Python :: 3',
src/datawrapper.pyx new file +201 -0
201 lines not on this page · raw message
src/trace2matrix.c new file +29 -0
@@ -0,0 +1,29 @@ +// SPDX-License-Identifier: LGPL-2.1 + +/* +* Copyright 2019 VMware Inc, Yordan Karadzhov <[email protected]> +*/ + +// KernelShark +#include "kernelshark/libkshark.h" + +ssize_t trace2matrix (uint64_t **offset_array, +uint16_t **cpu_array, +uint64_t **ts_array, +uint16_t **pid_array, +int **event_array) +{ +struct kshark_context *kshark_ctx =NULL ; +ssize_t total =0 ; + +if (!kshark_instance (&kshark_ctx)) +return -1 ; + + total =kshark_load_data_matrix (kshark_ctx, offset_array, + cpu_array, + ts_array, + pid_array, + event_array); + +return total; +}
Signature · 1 line
2.20.1
PATCH v2 4/12 trace-cruncher: Add "utils"
In reply to Yordan Karadzhov (VMware)Place the rest of the code, that is pure Python in tracecrunche/utils.py
--- tracecruncher/utils.py | 54 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 tracecruncher/utils.py
tracecruncher/utils.py new file +54 -0
@@ -0,0 +1,54 @@ +""" +SPDX-License-Identifier: LGPL-2.1 + +Copyright 2019 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> +""" + +import json + +from . import datawrapper as dw +from . import ksharkpy as ks + +def size(data): + """ Get the number of trace records. + """ + for key in dw.data_column_types: + if data[key] is not None: + return data[key].size + + raise Exception('Data size is unknown.') + +def save_session(session, s): + """ Save a KernelShark session description of a JSON file. + """ + s.seek(0) + json.dump(session, s, indent=4) + s.truncate() + + +def new_gui_session(fname, sname): + """ Generate and save a default KernelShark session description + file (JSON). + """ + ks.new_session_file(fname, sname) + + with open(sname, 'r+') as s: + session = json.load(s) + + session['Filters']['filter mask'] = 7 + session['CPUPlots'] = [] + session['TaskPlots'] = [] + session['Splitter'] = [1, 1] + session['MainWindow'] = [1200, 800] + session['ViewTop'] = 0 + session['ColorScheme'] = 0.75 + session['Model']['bins'] = 1000 + + session['Markers']['markA'] = {} + session['Markers']['markA']['isSet'] = False + session['Markers']['markB'] = {} + session['Markers']['markB']['isSet'] = False + session['Markers']['Active'] = 'A' + + save_session(session, s) +
Signature · 1 line
2.20.1
PATCH v2 5/12 trace-cruncher: Adapt sched_wakeup.py to use the new module
In reply to Yordan Karadzhov (VMware)--- examples/sched_wakeup.py | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
examples/sched_wakeup.py +15 -15
@@ -15,18 +15,18 @@ import matplotlib.pyplot as plt import scipy.stats as st import numpy as np -from ksharksetup import setup -# Always call setup() before importing ksharkpy!!! -setup() +import tracecruncher.datawrapper as dw +import tracecruncher.ksharkpy as ks +import tracecruncher.ftracepy as ft +import tracecruncher.utils as tc -import ksharkpy as ks # Get the name of the user program. if len(sys.argv) >= 2: fname = str(sys.argv[1]) else: fname = input('choose a trace file: ') -status = ks.open_file(fname) +status = ks.open(fname) if not status: print ("Failed to open file ", fname) sys.exit()
@@ -35,7 +35,7 @@ ks.register_plugin('reg_pid') # We do not need the Process Ids of the records. # Do not load the "pid" data. -data = ks.load_data(pid_data=False) +data = dw.load(pid_data=False) tasks = ks.get_tasks() # Get the name of the user program.
@@ -48,8 +48,8 @@ else: task_pid = tasks[prog_name][0] # Get the Event Ids of the sched_switch and sched_waking events. -ss_eid = ks.event_id('sched', 'sched_switch') -w_eid = ks.event_id('sched', 'sched_waking') +ss_eid = ft.event_id('sched', 'sched_switch') +w_eid = ft.event_id('sched', 'sched_waking') # Gey the size of the data. i = data['offset'].size
@@ -60,7 +60,7 @@ delta_max = i_ss_max = i_sw_max = 0 while i > 0: i = i - 1 if data['event'][i] == ss_eid: - next_pid = ks.read_event_field(offset=data['offset'][i], + next_pid = ft.read_event_field(offset=data['offset'][i], event_id=ss_eid, field='next_pid')
@@ -77,9 +77,9 @@ while i > 0: break if data['event'][i] == ss_eid: - next_pid = ks.read_event_field(offset=data['offset'][i], - event_id=ss_eid, - field='next_pid') + next_pid = ft.read_event_field(offset=data['offset'][i], + event_id=ss_eid, + field='next_pid') if next_pid == task_pid: # Second sched_switch for the same task. ? time_ss = data['time'][i]
@@ -89,7 +89,7 @@ while i > 0: continue if (data['event'][i] == w_eid): - waking_pid = ks.read_event_field(offset=data['offset'][i], + waking_pid = ft.read_event_field(offset=data['offset'][i], event_id=w_eid, field='pid')
@@ -120,7 +120,7 @@ ax.hist(dt, bins=(100), histtype='step') plt.show() sname = 'sched.json' -ks.new_session(fname, sname) +tc.new_gui_session(fname, sname) with open(sname, 'r+') as s: session = json.load(s)
@@ -143,6 +143,6 @@ with open(sname, 'r+') as s: session['ViewTop'] = int(i_sw_max) - 5 - ks.save_session(session, s) + tc.save_session(session, s) ks.close()
Signature · 1 line
2.20.1
PATCH v2 6/12 trace-cruncher: Add Makefile
In reply to Yordan Karadzhov (VMware)This will simplify the building procedure and will make it more intuitive.
--- Makefile | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 Makefile
Makefile new file +17 -0
@@ -0,0 +1,17 @@ +# +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2019 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> +# + +all: + python3 setup.py build + +clean: + rm -rf build src/datawrapper.c + +install: + python3 setup.py install --record install_manifest.txt + +uninstall: + xargs rm -v < install_manifest.txt
Signature · 1 line
2.20.1
PATCH v2 7/12 trace-cruncher: Adapt gpareto_fit.py to use the new module
In reply to Yordan Karadzhov (VMware)The patch contains some debugging as well.
--- examples/gpareto_fit.py | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-)
examples/gpareto_fit.py +17 -18
@@ -16,11 +16,10 @@ import numpy as np from scipy.stats import genpareto as gpareto from scipy.optimize import curve_fit as cfit -from ksharksetup import setup -# Always call setup() before importing ksharkpy!!! -setup() - -import ksharkpy as ks +import tracecruncher.datawrapper as dw +import tracecruncher.ksharkpy as ks +import tracecruncher.ftracepy as ft +import tracecruncher.utils as tc def chi2_test(hist, n_bins, c, loc, scale, norm): """ Simple Chi^2 test for the goodness of the fit.
@@ -92,7 +91,7 @@ def get_cpu_data(data, task_pid, start_id, stop_id, threshold): than the specified threshold. """ # Get the size of the data. - size = ks.data_size(data) + size = tc.size(data) #print("data size:", size) time_start = -1
@@ -130,7 +129,7 @@ def make_ks_session(fname, data, start, stop): The sessions is zooming around the maximum observed latency. """ sname = 'max_lat.json' - ks.new_session(fname, sname) + tc.new_gui_session(fname, sname) i_start = int(start) i_stop = int(stop)
@@ -145,28 +144,28 @@ def make_ks_session(fname, data, start, stop): session['Model']['range'] = [tmin, tmax] session['Markers']['markA']['isSet'] = True - session['Markers']['markA']['row'] = i_start) + session['Markers']['markA']['row'] = i_start session['Markers']['markB']['isSet'] = True - session['Markers']['markB']['row'] = i_stop) + session['Markers']['markB']['row'] = i_stop - session['ViewTop'] = i_start) - 5 + session['ViewTop'] = i_start - 5 - ks.save_session(session, s) + tc.save_session(session, s) fname = str(sys.argv[1]) -status = ks.open_file(fname) +status = ks.open(fname) if not status: print ("Failed to open file ", fname) sys.exit() ks.register_plugin('reg_pid') -data = ks.load_data() +data = dw.load() # Get the Event Ids of the hrtimer_start and print events. -start_id = ks.event_id('timer', 'hrtimer_start') -stop_id = ks.event_id('ftrace', 'print') +start_id = ft.event_id('timer', 'hrtimer_start') +stop_id = ft.event_id('ftrace', 'print') print("start_id", start_id) print("stop_id", stop_id)
@@ -194,9 +193,6 @@ ks.close() dt_ot = np.array(data_ot) np.savetxt('peak_over_threshold_loaded.txt', dt_ot) -make_ks_session(fname=fname, data=data, i_start=int(dt_ot[i_max_lat][1]), - i_stop=int(dt_ot[i_max_lat][2])) - P = len(dt_ot) / tot err_P = error_P(n=len(dt_ot), N=tot) print('tot:', tot, ' P =', P)
@@ -208,6 +204,9 @@ print('imax:', i_max_lat, int(dt_ot[i_max_lat][1])) print('max', np.amax(dt_ot)) +make_ks_session(fname=fname, data=data, start=dt_ot[i_max_lat][1], + stop=dt_ot[i_max_lat][2]) + start = threshold stop = 31 n_bins = (stop - start) * 2
Signature · 1 line
2.20.1
PATCH v2 8/12 trace-cruncher: Adapt page_faults.py to use the new module
In reply to Yordan Karadzhov (VMware)The patch contains some debugging as well.
--- examples/page_faults.py | 64 ++++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 36 deletions(-)
examples/page_faults.py +28 -36
@@ -9,20 +9,14 @@ Copyright 2019 VMware Inc, Yordan Karadzhov <[email protected]> import os import sys import subprocess as sp -import json - import pprint as pr -import matplotlib.pyplot as plt -import scipy.stats as st -import numpy as np from collections import Counter from tabulate import tabulate -from ksharksetup import setup -# Always call setup() before importing ksharkpy!!! -setup() - -import ksharkpy as ks +from tracecruncher import datawrapper as dw +from tracecruncher import ksharkpy as ks +from tracecruncher import ftracepy as ft +import tracecruncher.utils as tc def gdb_decode_address(obj_file, obj_address): """ Use gdb to examine the contents of the memory at this
@@ -33,7 +27,7 @@ def gdb_decode_address(obj_file, obj_address): '-ex', 'x/i ' + str(obj_address), obj_file], - stdout=sp.PIPE) + stdout=sp.PIPE) symbol = result.stdout.decode("utf-8").splitlines()
@@ -49,35 +43,35 @@ def gdb_decode_address(obj_file, obj_address): # Get the name of the tracing data file. fname = str(sys.argv[1]) -ks.open_file(fname) +ks.open(fname) ks.register_plugin('reg_pid') -data = ks.load_data() +data = dw.load() tasks = ks.get_tasks() #pr.pprint(tasks) # Get the Event Ids of the page_fault_user or page_fault_kernel events. -pf_eid = ks.event_id('exceptions', 'page_fault_user') +pf_eid = ft.event_id('exceptions', 'page_fault_user') # Gey the size of the data. -d_size = ks.data_size(data) +d_size = tc.size(data) # Get the name of the user program. prog_name = str(sys.argv[2]) table_headers = ['N p.f.', 'function', 'value', 'obj. file'] -table_list = [] # Loop over all tasks associated with the user program. for j in range(len(tasks[prog_name])): + table_list = [] count = Counter() task_pid = tasks[prog_name][j] for i in range(0, d_size): if data['event'][i] == pf_eid and data['pid'][i] == task_pid: - address = ks.read_event_field(offset=data['offset'][i], + address = ft.read_event_field(offset=data['offset'][i], event_id=pf_eid, field='address') - ip = ks.read_event_field(offset=data['offset'][i], + ip = ft.read_event_field(offset=data['offset'][i], event_id=pf_eid, field='ip') count[ip] += 1
@@ -92,29 +86,27 @@ for j in range(len(tasks[prog_name])): if i_max > len(pf_list): i_max = len(pf_list) + #print(pf_list[:25]) for i in range(0, i_max): - func = ks.get_function(pf_list[i][0]) - func_info = [func] - if func.startswith('0x'): - # The name of the function cannot be determined. We have an - # instruction pointer instead. Most probably this is a user-space - # function. - address = int(func, 0) - instruction = ks.map_instruction_address(task_pid, address) - - if instruction['obj_file'] != 'UNKNOWN': + func_info = [] + address = int(pf_list[i][0]) + func = ft.get_function(address) + if not func : + # The name of the function cannot be determined. Most probably + # this is a user-space function. + instruction = ft.map_instruction_address(pid=task_pid, + proc_addr=address) + if instruction: func_info = gdb_decode_address(instruction['obj_file'], instruction['address']) else: - func_info += ['', instruction['obj_file']] - - else: - func_info = [func] + func_info = ['addr: ' + hex(address), 'UNKNOWN'] table_list.append([pf_list[i][1]] + func_info) -ks.close() + print('\n{}-{}\n'.format(prog_name, task_pid), + tabulate(table_list, + headers=table_headers, + tablefmt='simple')) -print("\n", tabulate(table_list, - headers=table_headers, - tablefmt='simple')) +ks.close()
Signature · 1 line
2.20.1
PATCH v2 9/12 trace-cruncher: Automate the third-party build
In reply to Yordan Karadzhov (VMware)The goal is to provide a simple and intuitive installation procedure and in the same time to encapsulate the used third-party libraries inside the Python package. This way we avoid polluting the system-wide locations with the patched versions of the third-party libraries.
--- ...-_DEVEL-_LIBS-and-_RPATH_TO_ORIGIN-b.patch | 160 ++++++++++++++++++ 0001-kernel-shark-Add-_DEVEL-build-flag.patch | 90 ---------- 0002-kernel-shark-Add-reg_pid-plugin.patch | 8 +- Makefile | 4 +- install_third_party.sh | 39 +++++ setup.py | 32 ++-- src/ksharkpy.c | 4 +- tracecruncher/__init__.py | 4 + 8 files changed, 231 insertions(+), 110 deletions(-) create mode 100644 0001-kernel-shark-Add-_DEVEL-_LIBS-and-_RPATH_TO_ORIGIN-b.patch delete mode 100644 0001-kernel-shark-Add-_DEVEL-build-flag.patch create mode 100755 install_third_party.sh
0001-kernel-shark-Add-_DEVEL-_LIBS-and-_RPATH_TO_ORIGIN-b.patch new file +160 -0
@@ -0,0 +1,160 @@ +From 9e3a3cb07674b787f64d2f4ee9cb4467745e815b Mon Sep 17 00:00:00 2001 +From: "Yordan Karadzhov (VMware)" <[email protected]> +Date: Fri, 20 Sep 2019 14:31:15 +0300 +Subject: [PATCH 1/2] kernel-shark: Add _DEVEL, _LIBS and _RPATH_TO_ORIGIN + build flags + +If KernelShark is built with -D_LIBS=1 as a command-line argument +for Cmake, only libkshark and libkshapr-plot will be compiled. + +If KernelShark is built with -D_DEVEL=1 as a command-line argument +for Cmake, the headers of the libraries will be installed as well +and a symbolic link that points to the version of the library +being installed will be created. + +If built with -D_RPATH_TO_ORIGIN=1 the "rpath" of all libraries is +set to "$ORIGIN". + +Signed-off-by: Yordan Karadzhov (VMware) <[email protected]> +--- + kernel-shark/CMakeLists.txt | 23 ++++++++++++++------ + kernel-shark/README | 3 +++ + kernel-shark/src/CMakeLists.txt | 37 +++++++++++++++++++++++++++++++-- + 3 files changed, 55 insertions(+), 8 deletions(-) + +diff --git a/kernel-shark/CMakeLists.txt b/kernel-shark/CMakeLists.txt +index 20478b9..ea3133c 100644 +--- a/kernel-shark/CMakeLists.txt ++++ b/kernel-shark/CMakeLists.txt +@@ -73,6 +73,12 @@ else (CMAKE_BUILD_TYPE MATCHES Package) + + endif (CMAKE_BUILD_TYPE MATCHES Package) + ++if (_RPATH_TO_ORIGIN) ++ ++ set(CMAKE_INSTALL_RPATH $ORIGIN) ++ ++endif (_RPATH_TO_ORIGIN) ++ + include_directories(${KS_DIR}/src/ + ${KS_DIR}/build/src/ + ${JSONC_INCLUDE_DIR} +@@ -84,7 +90,6 @@ message(STATUS "CXX flags : " ${CMAKE_CXX_FLAGS}) + message(STATUS "Linker flags : " ${CMAKE_EXE_LINKER_FLAGS}) + + add_subdirectory(${KS_DIR}/src) +-add_subdirectory(${KS_DIR}/examples) + + if (_DOXYGEN_DOC AND DOXYGEN_FOUND) + +@@ -99,12 +104,18 @@ if (_DOXYGEN_DOC AND DOXYGEN_FOUND) + "${KS_DIR}/Documentation/doxygen/html" + "${KS_DIR}/Documentation/doxygen/latex") + +-endif () ++endif (_DOXYGEN_DOC AND DOXYGEN_FOUND) ++ ++if (NOT _LIBS) ++ ++ add_subdirectory(${KS_DIR}/examples) ++ ++ configure_file(${KS_DIR}/build/ks.desktop.cmake ++ ${KS_DIR}/${KS_APP_NAME}.desktop) + +-configure_file( ${KS_DIR}/build/ks.desktop.cmake +- ${KS_DIR}/${KS_APP_NAME}.desktop) ++ configure_file(${KS_DIR}/build/org.freedesktop.kshark-record.policy.cmake ++ ${KS_DIR}/org.freedesktop.kshark-record.policy) + +-configure_file( ${KS_DIR}/build/org.freedesktop.kshark-record.policy.cmake +- ${KS_DIR}/org.freedesktop.kshark-record.policy) ++endif (NOT _LIBS) + + message("") +diff --git a/kernel-shark/README b/kernel-shark/README +index 6c360bb..0f14212 100644 +--- a/kernel-shark/README ++++ b/kernel-shark/README +@@ -96,6 +96,9 @@ the dialog will derive the absolut path to the trace-cmd executable from + + If no build types is specified, the type will be "RelWithDebInfo". + ++2.1.4 In order to install a development version (including headers e.t.c) add ++-D_DEVEL=1 as a CMake Command-Line option. ++ + Examples: + + cmake -D_DOXYGEN_DOC=1 -D_INSTALL_PREFIX=/usr ../ +diff --git a/kernel-shark/src/CMakeLists.txt b/kernel-shark/src/CMakeLists.txt +index e20a030..f63da6a 100644 +--- a/kernel-shark/src/CMakeLists.txt ++++ b/kernel-shark/src/CMakeLists.txt +@@ -1,5 +1,13 @@ + message("\n src ...") + ++macro(install_symlink filepath sympath) ++ install(CODE "execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink ${filepath} ${sympath})") ++ install(CODE "LIST(APPEND CMAKE_INSTALL_MANIFEST_FILES ${sympath})") ++ install(CODE "message(\"-- Created symlink: ${sympath} -> ${filepath}\")") ++endmacro(install_symlink) ++ ++set(KS_INCLUDS_DESTINATION "${_INSTALL_PREFIX}/include/${KS_APP_NAME}") ++ + message(STATUS "libkshark") + add_library(kshark SHARED libkshark.c + libkshark-model.c +@@ -16,6 +24,19 @@ set_target_properties(kshark PROPERTIES SUFFIX ".so.${KS_VERSION_STRING}") + + install(TARGETS kshark LIBRARY DESTINATION ${_INSTALL_PREFIX}/lib/${KS_APP_NAME}) + ++if (_DEVEL) ++ ++ install_symlink("libkshark.so.${KS_VERSION_STRING}" ++ "${_INSTALL_PREFIX}/lib/${KS_APP_NAME}/libkshark.so") ++ ++ install(FILES "${KS_DIR}/src/libkshark.h" ++ "${KS_DIR}/src/libkshark-plugin.h" ++ "${KS_DIR}/src/libkshark-model.h" ++ DESTINATION ${KS_INCLUDS_DESTINATION} ++ COMPONENT devel) ++ ++endif (_DEVEL) ++ + if (OPENGL_FOUND AND GLUT_FOUND) + + message(STATUS "libkshark-plot") +@@ -30,9 +51,21 @@ if (OPENGL_FOUND AND GLUT_FOUND) + + install(TARGETS kshark-plot LIBRARY DESTINATION ${_INSTALL_PREFIX}/lib/${KS_APP_NAME}) + ++ if (_DEVEL) ++ ++ install_symlink("libkshark-plot.so.${KS_VERSION_STRING}" ++ "${_INSTALL_PREFIX}/lib/${KS_APP_NAME}/libkshark-plot.so") ++ ++ install(FILES "${KS_DIR}/src/KsPlotTools.hpp" ++ "${KS_DIR}/src/libkshark-plot.h" ++ DESTINATION ${KS_INCLUDS_DESTINATION} ++ COMPONENT devel) ++ ++ endif (_DEVEL) ++ + endif (OPENGL_FOUND AND GLUT_FOUND) + +-if (Qt5Widgets_FOUND AND Qt5Network_FOUND) ++if (Qt5Widgets_FOUND AND Qt5Network_FOUND AND NOT _LIBS) + + message(STATUS "libkshark-gui") + set (ks-guiLib_hdr KsUtils.hpp +@@ -98,7 +131,7 @@ if (Qt5Widgets_FOUND AND Qt5Network_FOUND) + install(PROGRAMS "${KS_DIR}/bin/kshark-su-record" + DESTINATION ${_INSTALL_PREFIX}/bin/) + +-endif (Qt5Widgets_FOUND AND Qt5Network_FOUND) ++endif (Qt5Widgets_FOUND AND Qt5Network_FOUND AND NOT _LIBS) + + add_subdirectory(plugins) + +-- +2.20.1 +
0001-kernel-shark-Add-_DEVEL-build-flag.patch deleted +0 -90
90 lines not on this page · raw message
0002-kernel-shark-Add-reg_pid-plugin.patch +4 -4
@@ -1,4 +1,4 @@ -From b3efcb6368bc7f70a23e156dce6c58d09953889a Mon Sep 17 00:00:00 2001 +From dbfb97e61e9907df195970cf378740a1ee927de9 Mon Sep 17 00:00:00 2001 From: "Yordan Karadzhov (VMware)" <[email protected]> Date: Wed, 9 Oct 2019 16:57:27 +0300 Subject: [PATCH 2/2] kernel-shark: Add "reg_pid" plugin
@@ -33,7 +33,7 @@ index 6c77179..bf69945 100644 set(PLUGINS ${PLUGIN_LIST} PARENT_SCOPE) diff --git a/kernel-shark/src/plugins/reg_pid.c b/kernel-shark/src/plugins/reg_pid.c new file mode 100644 -index 0000000..4116dd8 +index 0000000..df2e406 --- /dev/null +++ b/kernel-shark/src/plugins/reg_pid.c @@ -0,0 +1,189 @@
@@ -216,14 +216,14 @@ index 0000000..4116dd8 +/** Load this plugin. */ +int KSHARK_PLUGIN_INITIALIZER(struct kshark_context *kshark_ctx) +{ -+// printf("--> pid_reg init\n"); ++ printf("--> pid_reg init\n"); + return plugin_pid_reg_init(kshark_ctx); +} + +/** Unload this plugin. */ +int KSHARK_PLUGIN_DEINITIALIZER(struct kshark_context *kshark_ctx) +{ -+// printf("<-- pid reg close\n"); ++ printf("<-- pid reg close\n"); + return plugin_pid_reg_close(kshark_ctx); +} --
Makefile +3 -1
@@ -5,10 +5,12 @@ # all: + ./install_third_party.sh python3 setup.py build clean: - rm -rf build src/datawrapper.c + rm -rf third_party build tracecruncher/lib + rm -f src/datawrapper.c install: python3 setup.py install --record install_manifest.txt
install_third_party.sh new file +39 -0
@@ -0,0 +1,39 @@ +#!/bin/bash + +TRUNK=${PWD} +THIRD_PARTY_LIB=${TRUNK}/tracecruncher/lib/libtracecmd.so + +if [ -f ${THIRD_PARTY_LIB} ]; then + exit 0 +fi + +TAG=kernelshark-v1.1 +THIRD_PARTY_DIR=${TRUNK}/third_party +mkdir ${THIRD_PARTY_DIR} +cd ${THIRD_PARTY_DIR} + +echo 'Installing: ' ${TAG} + +git clone git://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git --branch=${TAG} + +cd trace-cmd +git am ${TRUNK}/0001-kernel-shark-* +git am ${TRUNK}/0002-kernel-shark-* + +make prefix=${THIRD_PARTY_DIR} install_libs + +cd kernel-shark/build/ +cmake -D_DEVEL=1 -D_LIBS=1 -D_INSTALL_PREFIX=${THIRD_PARTY_DIR} -D_RPATH_TO_ORIGIN=1 .. +make install + +cd ${TRUNK} + +LIB_DIR=${TRUNK}/tracecruncher/lib/ +if [ ! -d "${LIB_DIR}" ]; then + mkdir ${LIB_DIR} +fi + +cp -v ${THIRD_PARTY_DIR}/lib/kernelshark/libkshark.so.* ${TRUNK}/tracecruncher/lib/ +cp -v ${THIRD_PARTY_DIR}/lib/kernelshark/plugins/*.so ${TRUNK}/tracecruncher/lib/ +cp -v ${THIRD_PARTY_DIR}/lib/traceevent/*.so ${TRUNK}/tracecruncher/lib/ +cp -v ${THIRD_PARTY_DIR}/lib/trace-cmd/*.so ${TRUNK}/tracecruncher/lib/
setup.py +18 -14
@@ -12,34 +12,37 @@ from Cython.Build import cythonize import numpy as np def main(): - kshark_path = '/usr/local/lib/kernelshark' - traceevent_path = '/usr/local/lib/traceevent/' - tracecmd_path = '/usr/local/lib/trace-cmd/' + third_party = './third_party' cythonize('src/datawrapper.pyx') + + third_party_libdirs = [third_party+'/lib/kernelshark', + third_party+'/lib/traceevent', + third_party+'/lib/trace-cmd'] + + runtime_library_dirs=['$ORIGIN/lib'] + module_data = Extension('tracecruncher.datawrapper', sources=['src/datawrapper.c'], - include_dirs=[np.get_include()], - library_dirs=[kshark_path, traceevent_path, tracecmd_path], - runtime_library_dirs=[kshark_path, traceevent_path, tracecmd_path], + include_dirs=[np.get_include(), third_party+'/include'], + library_dirs=third_party_libdirs, + runtime_library_dirs=runtime_library_dirs, libraries=['kshark', 'traceevent', 'tracecmd'] ) module_ks = Extension('tracecruncher.ksharkpy', sources=['src/ksharkpy.c'], - library_dirs=[kshark_path], - runtime_library_dirs=[kshark_path], + include_dirs=[third_party+'/include'], + library_dirs=third_party_libdirs, + runtime_library_dirs=runtime_library_dirs, libraries=['kshark'], - define_macros=[ - ('LIB_KSHARK_PATH', '\"' + kshark_path + '/libkshark.so\"'), - ('KS_PLUGIN_DIR', '\"' + kshark_path + '/plugins\"') - ], ) module_ft = Extension('tracecruncher.ftracepy', sources=['src/ftracepy.c'], - library_dirs=[kshark_path, traceevent_path, tracecmd_path], - runtime_library_dirs=[kshark_path, traceevent_path, tracecmd_path], + include_dirs=[third_party+'/include'], + library_dirs=third_party_libdirs, + runtime_library_dirs=runtime_library_dirs, libraries=['kshark', 'traceevent', 'tracecmd'], )
@@ -52,6 +55,7 @@ def main(): license='LGPL-2.1', packages=find_packages(), ext_modules=[module_data, module_ks, module_ft], + package_data={'tracecruncher': ['lib/*.so*']}, classifiers=[ 'Development Status :: 3 - Alpha', 'Programming Language :: Python :: 3',
src/ksharkpy.c +3 -1
@@ -137,7 +137,9 @@ static PyObject *method_register_plugin(PyObject *self, PyObject *args,return NULL ; } -if (asprintf (&lib_file,"%s/plugin-%s.so" ,KS_PLUGIN_DIR , plugin) <0 ) { +if (asprintf (&lib_file,"%s/lib/plugin-%s.so" , +getenv ("TRACE_CRUNCHER_PATH" ), + plugin) <0 ) {KS_MEM_ERROR return NULL; }
tracecruncher/__init__.py +4 -0
@@ -0,0 +1,4 @@ +import os + +pwd = os.path.dirname(os.path.abspath(__file__)) +os.environ['TRACE_CRUNCHER_PATH'] = pwd
Signature · 1 line
2.20.1
PATCH v2 10/12 trace-cruncher: Update README.md
In reply to Yordan Karadzhov (VMware)--- README.md | 50 +++++++++++--------------------------------------- 1 file changed, 11 insertions(+), 39 deletions(-)
README.md +11 -39
@@ -4,9 +4,9 @@ ## Overview -The Trace-Cruncher project aims to provide an interface between the existing instrumentation for collection and visualization of tracing data from the Linux kernel and the broad and very well developed ecosystem of instruments for data analysis available in Python. The interface will be based on NumPy. +The Trace-Cruncher project aims to provide an interface between the existing instrumentation for collection and visualization of tracing data from the Linux kernel and the broad and very well developed ecosystem of instruments for data analysis available in Python. The interface is based on NumPy. -NumPy implements an efficient multi-dimensional container of generic data and uses strong typing in order to provide fast data processing in Python. The Trace-Cruncher will allow for sophisticated analysis of kernel tracing data via scripts, but it will also opens the door for exposing the kernel tracing data to the instruments provided by the scientific toolkit of Python like MatPlotLib, Stats, Scikit-Learn and even to the nowadays most popular frameworks for Machine Learning like TensorFlow and PyTorch. The Trace-Cruncher is strongly coupled to the KernelShark project and is build on top of the C API of libkshark. +NumPy implements an efficient multi-dimensional container of generic data and uses strong typing in order to provide fast data processing in Python. The Trace-Cruncher allows for sophisticated analysis of kernel tracing data via scripts, but it also opens the door for exposing the kernel tracing data to the instruments provided by the scientific toolkit of Python like MatPlotLib, Stats, Scikit-Learn and even to the nowadays most popular frameworks for Machine Learning like TensorFlow and PyTorch. The Trace-Cruncher is strongly coupled to the KernelShark project and is build on top of the C API of libkshark. ## Try it out
@@ -17,55 +17,27 @@ Trace-Cruncher has the following external dependencies: 1.1 In order to install the packages on Ubuntu do the following: - sudo apt-get install libjson-c-dev libpython3-dev cython3 -y + > sudo apt-get install libjson-c-dev libpython3-dev cython3 -y - sudo apt-get install python3-numpy python3-matplotlib -y + > sudo apt-get install python3-numpy python3-matplotlib -y 1.2 In order to install the packages on Fedora, as root do the following: - dnf install json-c-devel python3-devel python3-Cython -y + > dnf install json-c-devel python3-devel python3-Cython -y - dnf install python3-numpy python3-matplotlib -y - -2. In order to get the proper version of KernelShark / trace-cmd do the -following: - - git clone git://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git --branch=kernelshark-v1.1 - -or download a tarball from here: -https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/snapshot/trace-cmd-kernelshark-v1.1.tar.gz + > dnf install python3-numpy python3-matplotlib -y ### Build & Run -1. Patch trace-cmd / KernelShark: - - cd path/to/trace-cmd/ - - git am ../path/to/trace-cruncher/0001-kernel-shark-Add-_DEVEL-build-flag.patch - - git am ../path/to/trace-cruncher/0002-kernel-shark-Add-reg_pid-plugin.patch - -2. Install trace-cmd: - - make - - sudo make install_libs - -3. Install KernelShark: - - cd kernel-shark/build - - cmake -D_DEVEL=1 ../ - - make +Installing trace-cruncher is very simple. After downloading the source code, you just have to run: - sudo make install + > cd trace-cruncher -4. Build the NumPy API itself: + > make - cd path/to/trace-cruncher + > sudo make install - ./np_setup.py build_ext -i +Note that this will automatically download, patch and build the appropriate versions of "trace-cmd / KernelShark" libraries from kernel.org. These third-party libraries will be installed as part of trace-cruncher itself and will not interfere with any existing system-wide installations of trace-cmd and KernelShark. ## Documentation
Signature · 1 line
2.20.1
PATCH v2 11/12 trace-cruncher: Remove all leftover files.
In reply to Yordan Karadzhov (VMware)This patch completes the refactoring of trace-cruncher into a Python module. All obsoleted source files are removed.
--- clean.sh | 6 - examples/ksharksetup.py | 24 --- libkshark-py.c | 224 ------------------------- libkshark_wrapper.pyx | 361 ---------------------------------------- np_setup.py | 90 ---------- 5 files changed, 705 deletions(-) delete mode 100755 clean.sh delete mode 100644 examples/ksharksetup.py delete mode 100644 libkshark-py.c delete mode 100644 libkshark_wrapper.pyx delete mode 100755 np_setup.py
clean.sh deleted +0 -6
@@ -1,6 +0,0 @@ -#!/bin/bash - -rm libkshark_wrapper.c -rm ksharkpy.cpython-3*.so -rm -rf build/ -rm -rf examples/__pycache__/
examples/ksharksetup.py deleted +0 -24
@@ -1,24 +0,0 @@ -#!/usr/bin/env python3 - -""" -SPDX-License-Identifier: LGPL-2.1 - -Copyright 2019 VMware Inc, Yordan Karadzhov <[email protected]> -""" - -import os -import sys - -def setup(): - os.chdir(os.path.dirname(__file__)) - path = os.getcwd() + '/..' - sys.path.append(path) - - if 'LD_LIBRARY_PATH' not in os.environ: - os.environ['LD_LIBRARY_PATH'] = '/usr/local/lib/kernelshark:/usr/local/lib/traceevent:/usr/local/lib/trace-cmd' - try: - os.execv(sys.argv[0], sys.argv) - except e: - print('Failed re-exec:', e) - sys.exit(1) -
libkshark-py.c deleted +0 -224
224 lines not on this page · raw message
libkshark_wrapper.pyx deleted +0 -361
361 lines not on this page · raw message
np_setup.py deleted +0 -90
90 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH v2 12/12 trace-cruncher: Improve Makefile Provide more robust and better looking build process.
In reply to Yordan Karadzhov (VMware)--- Makefile | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
Makefile +18 -1
@@ -4,8 +4,17 @@ # Copyright 2019 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> # +UID := $(shell id -u) + +CYAN := '\e[36m' +PURPLE := '\e[35m' +NC := '\e[0m' + all: + @ echo ${CYAN}Installing third party:${NC}; ./install_third_party.sh + @ echo + @ echo ${CYAN}Buildinging trace-cruncher:${NC}; python3 setup.py build clean:
@@ -13,7 +22,15 @@ clean: rm -f src/datawrapper.c install: + @ echo ${CYAN}Installing trace-cruncher:${NC}; python3 setup.py install --record install_manifest.txt uninstall: - xargs rm -v < install_manifest.txt + @ if [ $(UID) -ne 0 ]; then \ + echo ${PURPLE}Permission denied${NC} 1>&2; \ + else \ + echo ${CYAN}Uninstalling trace-cruncher:${NC}; \ + xargs rm -v < install_manifest.txt; \ + rm -rfv dist tracecruncher.egg-info; \ + rm -fv install_manifest.txt; \ + fi
Signature · 1 line
2.20.1