Build trace-cruncher as Python pakage
v2 cover Build trace-cruncher as Python pakage
This patch-set restructures the project and makes it build as a native Python package. Although it looks like a complete rewrite, this is essentially just a switching from using Cython to using directly the C API of Python. Cython is still being used but only for the implementation of the NumPy data wrapper. The functionalities that are wrapping Ftrace are extended substantially. This is possible due to switching to use of the recently released libraries: libtraceevent and libtracefs.
Changes in v2:
- Addressing the comments made by Steven in his review.
- Start using the libtracefs APIs for enable/disable events.
- Add functionalities for enable/disable event filters.
Yordan Karadzhov (VMware) (9):
trace-cruncher: Refactor the part that wraps ftrace
trace-cruncher: Refactor the part that wraps libkshark
trace-cruncher: Add "utils"
trace-cruncher: Refactor the examples
trace-cruncher: Add Makefile
trace-cruncher: Update README.md
trace-cruncher: Remove all leftover files.
trace-cruncher: Add testing
trace-cruncher: Add github workflow for CI testing
.github/workflows/main.yml | 58 + 0001-kernel-shark-Add-_DEVEL-build-flag.patch | 90 - 0002-kernel-shark-Add-reg_pid-plugin.patch | 231 --- Makefile | 33 + README.md | 84 +- clean.sh | 6 - examples/gpareto_fit.py | 328 ---- examples/ksharksetup.py | 24 - examples/page_faults.py | 120 -- examples/sched_wakeup.py | 70 +- libkshark-py.c | 224 --- libkshark_wrapper.pyx | 361 ---- np_setup.py | 90 - setup.py | 81 + src/common.h | 105 + src/ftracepy-utils.c | 1712 +++++++++++++++++ src/ftracepy-utils.h | 144 ++ src/ftracepy.c | 292 +++ src/ksharkpy-utils.c | 411 ++++ src/ksharkpy-utils.h | 41 + src/ksharkpy.c | 94 + src/npdatawrapper.pyx | 203 ++ src/trace2matrix.c | 40 + tests/0_get_data/__init__.py | 0 tests/0_get_data/test_get_data.py | 26 + tests/1_unit/__init__.py | 0 tests/1_unit/test_01_ftracepy_unit.py | 495 +++++ tests/1_unit/test_02_datawrapper_unit.py | 41 + tests/1_unit/test_03_ksharkpy_unit.py | 72 + tests/2_integration/__init__.py | 0 .../test_01_ftracepy_integration.py | 113 ++ .../test_03_ksharkpy_integration.py | 25 + tests/__init__.py | 0 tracecruncher/__init__.py | 0 tracecruncher/ft_utils.py | 28 + tracecruncher/ks_utils.py | 227 +++ 36 files changed, 4325 insertions(+), 1544 deletions(-) create mode 100644 .github/workflows/main.yml delete mode 100644 0001-kernel-shark-Add-_DEVEL-build-flag.patch delete mode 100644 0002-kernel-shark-Add-reg_pid-plugin.patch create mode 100644 Makefile delete mode 100755 clean.sh delete mode 100755 examples/gpareto_fit.py delete mode 100644 examples/ksharksetup.py delete mode 100755 examples/page_faults.py delete mode 100644 libkshark-py.c delete mode 100644 libkshark_wrapper.pyx delete mode 100755 np_setup.py create mode 100644 setup.py create mode 100644 src/common.h create mode 100644 src/ftracepy-utils.c create mode 100644 src/ftracepy-utils.h create mode 100644 src/ftracepy.c create mode 100644 src/ksharkpy-utils.c create mode 100644 src/ksharkpy-utils.h create mode 100644 src/ksharkpy.c create mode 100644 src/npdatawrapper.pyx create mode 100644 src/trace2matrix.c create mode 100644 tests/0_get_data/__init__.py create mode 100755 tests/0_get_data/test_get_data.py create mode 100644 tests/1_unit/__init__.py create mode 100644 tests/1_unit/test_01_ftracepy_unit.py create mode 100755 tests/1_unit/test_02_datawrapper_unit.py create mode 100755 tests/1_unit/test_03_ksharkpy_unit.py create mode 100644 tests/2_integration/__init__.py create mode 100755 tests/2_integration/test_01_ftracepy_integration.py create mode 100755 tests/2_integration/test_03_ksharkpy_integration.py create mode 100644 tests/__init__.py create mode 100644 tracecruncher/__init__.py create mode 100644 tracecruncher/ft_utils.py create mode 100644 tracecruncher/ks_utils.py
Signature · 1 line
2.27.0
PATCH v2 1/9 trace-cruncher: Refactor the part that wraps ftrace
In order to be able to bulid the project as a native Python package, which contains several sub-packages implement as C extensions via the Python's C API, the part of the interface that relies on libtracefs, libtraceevent (and libtracecmd in the future) needs to be re-implemented as an extension called "tracecruncher.ftracepy". Note that this new extension has a stand-alone build that is completely decoupled from the existing build system used by trace-cruncher.
--- setup.py | 68 ++ src/common.h | 105 +++ src/ftracepy-utils.c | 1712 ++++++++++++++++++++++++++++++++++++++++++ src/ftracepy-utils.h | 144 ++++ src/ftracepy.c | 292 +++++++ 5 files changed, 2321 insertions(+) create mode 100644 setup.py create mode 100644 src/common.h create mode 100644 src/ftracepy-utils.c create mode 100644 src/ftracepy-utils.h create mode 100644 src/ftracepy.c
setup.py new file +68 -0
@@ -0,0 +1,68 @@ +#!/usr/bin/env python3 + +""" +SPDX-License-Identifier: LGPL-2.1 + +Copyright 2019 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> +""" + +from setuptools import setup, find_packages +from distutils.core import Extension +from Cython.Build import cythonize + +import pkgconfig as pkg + + +def third_party_paths(): + pkg_traceevent = pkg.parse('libtraceevent') + pkg_ftracepy = pkg.parse('libtracefs') + pkg_tracecmd = pkg.parse('libtracecmd') + + include_dirs = [] + include_dirs.extend(pkg_traceevent['include_dirs']) + include_dirs.extend(pkg_ftracepy['include_dirs']) + include_dirs.extend(pkg_tracecmd['include_dirs']) + + library_dirs = [] + library_dirs.extend(pkg_traceevent['library_dirs']) + library_dirs.extend(pkg_ftracepy['library_dirs']) + library_dirs.extend(pkg_tracecmd['library_dirs']) + library_dirs = list(set(library_dirs)) + + return include_dirs, library_dirs + +include_dirs, library_dirs = third_party_paths() + +def extension(name, sources, libraries): + runtime_library_dirs = library_dirs + runtime_library_dirs.extend('$ORIGIN') + return Extension(name, sources=sources, + include_dirs=include_dirs, + library_dirs=library_dirs, + runtime_library_dirs=runtime_library_dirs, + libraries=libraries, + ) + +def main(): + module_ft = extension(name='tracecruncher.ftracepy', + sources=['src/ftracepy.c', 'src/ftracepy-utils.c'], + libraries=['traceevent', 'tracefs']) + + setup(name='tracecruncher', + version='0.1.0', + description='NumPy based interface for accessing tracing data in Python.', + author='Yordan Karadzhov (VMware)', + author_email='[email protected]', + url='https://github.com/vmware/trace-cruncher', + license='LGPL-2.1', + packages=find_packages(), + ext_modules=[module_ft], + classifiers=[ + 'Development Status :: 3 - Alpha', + 'Programming Language :: Python :: 3', + ] + ) + + +if __name__ == '__main__': + main()
src/common.h new file +105 -0
105 lines not on this page · raw message
src/ftracepy-utils.c new file +1712 -0
1,712 lines not on this page · raw message
src/ftracepy-utils.h new file +144 -0
144 lines not on this page · raw message
src/ftracepy.c new file +292 -0
292 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH v2 2/9 trace-cruncher: Refactor the part that wraps libkshark
In reply to Yordan Karadzhov (VMware)The part of the interface that relies on libkshark gets re-implemented as an extension called "tracecruncher.ksharkpy". The new extension gets build together with the previously implemented "tracecruncher.ftracefy" extension.
--- setup.py | 17 +- src/ksharkpy-utils.c | 411 ++++++++++++++++++++++++++++++++++++++++++ src/ksharkpy-utils.h | 41 +++++ src/ksharkpy.c | 94 ++++++++++ src/npdatawrapper.pyx | 203 +++++++++++++++++++++ src/trace2matrix.c | 40 ++++ 6 files changed, 804 insertions(+), 2 deletions(-) create mode 100644 src/ksharkpy-utils.c create mode 100644 src/ksharkpy-utils.h create mode 100644 src/ksharkpy.c create mode 100644 src/npdatawrapper.pyx create mode 100644 src/trace2matrix.c
setup.py +15 -2
@@ -11,22 +11,26 @@ from distutils.core import Extension from Cython.Build import cythonize import pkgconfig as pkg +import numpy as np def third_party_paths(): pkg_traceevent = pkg.parse('libtraceevent') pkg_ftracepy = pkg.parse('libtracefs') pkg_tracecmd = pkg.parse('libtracecmd') + pkg_kshark = pkg.parse('libkshark') - include_dirs = [] + include_dirs = [np.get_include()] include_dirs.extend(pkg_traceevent['include_dirs']) include_dirs.extend(pkg_ftracepy['include_dirs']) include_dirs.extend(pkg_tracecmd['include_dirs']) + include_dirs.extend(pkg_kshark['include_dirs']) library_dirs = [] library_dirs.extend(pkg_traceevent['library_dirs']) library_dirs.extend(pkg_ftracepy['library_dirs']) library_dirs.extend(pkg_tracecmd['library_dirs']) + library_dirs.extend(pkg_kshark['library_dirs']) library_dirs = list(set(library_dirs)) return include_dirs, library_dirs
@@ -48,6 +52,15 @@ def main(): sources=['src/ftracepy.c', 'src/ftracepy-utils.c'], libraries=['traceevent', 'tracefs']) + cythonize('src/npdatawrapper.pyx', language_level = "3") + module_data = extension(name='tracecruncher.npdatawrapper', + sources=['src/npdatawrapper.c'], + libraries=['kshark']) + + module_ks = extension(name='tracecruncher.ksharkpy', + sources=['src/ksharkpy.c', 'src/ksharkpy-utils.c'], + libraries=['kshark']) + setup(name='tracecruncher', version='0.1.0', description='NumPy based interface for accessing tracing data in Python.',
@@ -56,7 +69,7 @@ def main(): url='https://github.com/vmware/trace-cruncher', license='LGPL-2.1', packages=find_packages(), - ext_modules=[module_ft], + ext_modules=[module_ft, module_data, module_ks], classifiers=[ 'Development Status :: 3 - Alpha', 'Programming Language :: Python :: 3',
src/ksharkpy-utils.c new file +411 -0
411 lines not on this page · raw message
src/ksharkpy-utils.h new file +41 -0
41 lines not on this page · raw message
src/ksharkpy.c new file +94 -0
94 lines not on this page · raw message
src/npdatawrapper.pyx new file +203 -0
203 lines not on this page · raw message
src/trace2matrix.c new file +40 -0
@@ -0,0 +1,40 @@ +// SPDX-License-Identifier: LGPL-2.1 + +/* +* Copyright 2019 VMware Inc, Yordan Karadzhov <[email protected]> +*/ + +// KernelShark +#include "libkshark.h" + +ssize_t trace2matrix (int sd, +int16_t **event_array, +int16_t **cpu_array, +int32_t **pid_array, +int64_t **offset_array, +int64_t **ts_array) +{ +struct kshark_generic_stream_interface *interface; +struct kshark_context *kshark_ctx =NULL ; +struct kshark_data_stream *stream; +ssize_t total =0 ; + +if (!kshark_instance (&kshark_ctx)) +return -1 ; + + stream =kshark_get_data_stream (kshark_ctx, sd); +if (!stream) +return -1 ; + + interface = stream->interface; +if (interface->type ==KS_GENERIC_DATA_INTERFACE && + interface->load_matrix) { + total = interface->load_matrix (stream, kshark_ctx, event_array, + cpu_array, + pid_array, + offset_array, + ts_array); + } + +return total; +}
Signature · 1 line
2.27.0
PATCH v2 3/9 trace-cruncher: Add "utils"
In reply to Yordan Karadzhov (VMware)Place all the code, that is pure Python, in tracecrunche/ks_utils.py and tracecrunche/ft_utils.py
--- tracecruncher/__init__.py | 0 tracecruncher/ft_utils.py | 28 +++++ tracecruncher/ks_utils.py | 227 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 255 insertions(+) create mode 100644 tracecruncher/__init__.py create mode 100644 tracecruncher/ft_utils.py create mode 100644 tracecruncher/ks_utils.py
tracecruncher/__init__.py new file +0 -0
tracecruncher/ft_utils.py new file +28 -0
@@ -0,0 +1,28 @@ +""" +SPDX-License-Identifier: LGPL-2.1 + +Copyright 2019 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> +""" + +import sys +import time +import hashlib + +from . import ftracepy as ft + + +def find_event_id(system, event): + """ Get the unique identifier of a trace event. + """ + tep = ft.tep_handle(); + tep.init_local(dir=ft.dir(), systems=[system]); + + return tep.get_event(system=system, name=event).id() + +def instance_autoname(): + """ Generates a quasi-random (unique) name for a trace instance. + """ + unique = ''.join(sys.argv) + str(time.time()) + hash = hashlib.sha1() + hash.update(unique.encode('utf-8')) + return hash.hexdigest()
tracecruncher/ks_utils.py new file +227 -0
227 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH v2 4/9 trace-cruncher: Refactor the examples
In reply to Yordan Karadzhov (VMware)For the moment we will keep only one example that is "sched_wakeup.py". "gpareto_fit.py" gets removed because it doesn't really demonstrate anything conceptually different compared to "sched_wakeup.py". The difference comes from the more advanced statistical analysis of the data, however this goes beyond the scope of trace-cruncher. "page_faults.py" gets removed only temporally, because it requires some functionalities that are not yet implemented in the ftrace libraries. Once those functionalities are made available, the example will be added to trace-cruncher again.
--- examples/gpareto_fit.py | 328 --------------------------------------- examples/page_faults.py | 120 -------------- examples/sched_wakeup.py | 70 ++++----- 3 files changed, 30 insertions(+), 488 deletions(-) delete mode 100755 examples/gpareto_fit.py delete mode 100755 examples/page_faults.py
examples/gpareto_fit.py deleted +0 -328
@@ -1,328 +0,0 @@ -#!/usr/bin/env python3 - -""" -SPDX-License-Identifier: LGPL-2.1 - -Copyright 2019 VMware Inc, Yordan Karadzhov <[email protected]> -""" - -import sys -import json - -import matplotlib.pyplot as plt -import scipy.stats as st -import numpy as np - -from scipy.stats import genpareto as gpareto -from scipy.optimize import curve_fit as cfit - -from ksharksetup import setup -# Always call setup() before importing ksharkpy!!! -setup() - -import ksharkpy as ks - -def chi2_test(hist, n_bins, c, loc, scale, norm): - """ Simple Chi^2 test for the goodness of the fit. - """ - chi2 = n_empty_bins = 0 - for i in range(len(hist[0])): - if hist[0][i] == 0: - # Ignore this empty bin. - n_empty_bins += 1 - continue - - # Get the center of bin i. - x = (hist[1][i] + hist[1][i + 1]) / 2 - fit_val = gpareto.pdf(x, c=c, loc=loc, scale=scale) - chi = (fit_val - hist[0][i]) / np.sqrt(hist[0][i]) - chi2 += chi**2 - - return norm * chi2 / (n_bins - n_empty_bins) - -def quantile(p, P, c, loc, scale): - """ The quantile function of the Generalized Pareto distribution. - """ - return loc + scale / c * ((P / p)**(c) - 1) - - -def dq_dscale(p, P, c, scale): - """ Partial derivative of the quantile function. - """ - return ((P / p)**c - 1) / c - - -def dq_dc(p, P, c, scale): - """ Partial derivative of the quantile function. - """ - return (scale * (np.log(P / p) * (P / p)**c ) / c - - scale * ((P / p)**c - 1) / (c**2)) - - -def dq_dP(p, P, c, scale): - """ Partial derivative of the quantile function. - """ - return scale / P * (P / p)**c - - -def error_P(n, N): - return np.sqrt(n) / N - - -def error(p, P, c, scale, err_P, err_c, err_scale): - return np.sqrt((dq_dP(p, P, c, scale) * err_P)**2 - + (dq_dc(p, P, c, scale) * err_c)**2 - + (dq_dscale(p, P, c, scale) * err_scale)**2) - - -def quantile_conf_bound(p, P, n, c, loc, scale, err_P, err_c, err_scale): - return (quantile(p=p, P=P, c=c, loc=loc, scale=scale) - + n * error(p=p, P=P, c=c, scale=scale, - err_P=err_P, err_c=err_c, err_scale=err_scale)); - - -def get_latency(t0, t1): - """ Get the value of the latency in microseconds - """ - return (t1 - t0) / 1000 - 1000 - - -def get_cpu_data(data, task_pid, start_id, stop_id, threshold): - """ Loop over the tracing data for a given CPU and find all latencies bigger - than the specified threshold. - """ - # Get the size of the data. - size = ks.data_size(data) - #print("data size:", size) - - time_start = -1 - dt_ot = [] - tot = 0 - i = 0 - i_start = 0; - - while i < size: - if data["pid"][i] == task_pid and data['event'][i] == start_id: - time_start = data['time'][i] - i_start = i; - i = i + 1 - - while i < size: - if data["pid"][i] == task_pid and data['event'][i] == stop_id: - delta = get_latency(time_start, data['time'][i]) - - if delta > threshold and tot != 0: - print('lat. over threshold: ', delta, i_start, i) - dt_ot.append([delta, i_start, i]) - - tot = tot + 1 - break - - i = i + 1 - i = i + 1 - - print(task_pid, 'tot:', len(dt_ot), '/', tot) - return dt_ot, tot - - -def make_ks_session(fname, data, start, stop): - """ Save a KernelShark session descriptor file (Json). - The sessions is zooming around the maximum observed latency. - """ - sname = 'max_lat.json' - ks.new_session(fname, sname) - i_start = int(start) - i_stop = int(stop) - - with open(sname, 'r+') as s: - session = json.load(s) - session['TaskPlots'] = [int(data['pid'][i_start])] - session['CPUPlots'] = [int(data['cpu'][i_start])] - - delta = data['time'][i_stop] - data['time'][i_start] - tmin = int(data['time'][i_start] - delta) - tmax = int(data['time'][i_stop] + delta) - session['Model']['range'] = [tmin, tmax] - - session['Markers']['markA']['isSet'] = True - session['Markers']['markA']['row'] = i_start) - - session['Markers']['markB']['isSet'] = True - session['Markers']['markB']['row'] = i_stop) - - session['ViewTop'] = i_start) - 5 - - ks.save_session(session, s) - - -fname = str(sys.argv[1]) -status = ks.open_file(fname) -if not status: - print ("Failed to open file ", fname) - sys.exit() - -ks.register_plugin('reg_pid') -data = ks.load_data() - -# Get the Event Ids of the hrtimer_start and print events. -start_id = ks.event_id('timer', 'hrtimer_start') -stop_id = ks.event_id('ftrace', 'print') -print("start_id", start_id) -print("stop_id", stop_id) - -tasks = ks.get_tasks() -jdb_pids = tasks['jitterdebugger'] -print('jitterdeburrer pids:', jdb_pids) -jdb_pids.pop(0) - -threshold = 10 -data_ot = [] -tot = 0 - -for task_pid in jdb_pids: - cpu_data, cpu_tot = get_cpu_data(data=data, - task_pid=task_pid, - start_id=start_id, - stop_id=stop_id, - threshold=threshold) - - data_ot.extend(cpu_data) - tot += cpu_tot - -ks.close() - -dt_ot = np.array(data_ot) -np.savetxt('peak_over_threshold_loaded.txt', dt_ot) - -make_ks_session(fname=fname, data=data, i_start=int(dt_ot[i_max_lat][1]), - i_stop=int(dt_ot[i_max_lat][2])) - -P = len(dt_ot) / tot -err_P = error_P(n=len(dt_ot), N=tot) -print('tot:', tot, ' P =', P) - -lat = dt_ot[:,0] -#print(lat) -i_max_lat = lat.argmax() -print('imax:', i_max_lat, int(dt_ot[i_max_lat][1])) - -print('max', np.amax(dt_ot)) - -start = threshold -stop = 31 -n_bins = (stop - start) * 2 - -bin_size = (stop - start) / n_bins - -x = np.linspace(start=start + bin_size / 2, - stop=stop - bin_size / 2, - num=n_bins) - -bins_ot = np.linspace(start=start, stop=stop, num=n_bins + 1) -#print(bins_ot) - -fig, ax = plt.subplots(nrows=2, ncols=2) -fig.tight_layout() -ax[-1, -1].axis('off') - -hist_ot = ax[0][0].hist(x=lat, bins=bins_ot, histtype='stepfilled', alpha=0.3) -ax[0][0].set_xlabel('latency [\u03BCs]', fontsize=8) -ax[0][0].set_yscale('log') -#print(hist_ot[0]) - -hist_ot_norm = ax[1][0].hist(x=lat, bins=bins_ot, - density=True, histtype='stepfilled', alpha=0.3) - -# Fit using the fitter of the genpareto class (shown in red). -ret = gpareto.fit(lat, loc=threshold) -ax[1][0].plot(x, gpareto.pdf(x, c=ret[0], loc=ret[1], scale=ret[2]), - 'r-', lw=1, color='red', alpha=0.8) - -ax[1][0].set_xlabel('latency [\u03BCs]', fontsize=8) -print(ret) -print('\ngoodness-of-fit: ' + '{:03.3f}'.format(chi2_test(hist_ot_norm, - n_bins=n_bins, - c=ret[0], - loc=ret[1], - scale=ret[2], - norm=len(lat)))) - -print("\n curve_fit:") -# Fit using the curve_fit fitter. Fix the value of the "loc" parameter. -popt, pcov = cfit(lambda x, c, scale: gpareto.pdf(x, c=c, loc=threshold, scale=scale), - x, hist_ot_norm[0], - p0=[ret[0], ret[2]]) - -print(popt) -print(pcov) - -ax[1][0].plot(x, gpareto.pdf(x, c=popt[0], loc=threshold, scale=popt[1]), - 'r-', lw=1, color='blue', alpha=0.8) - -fit_legend = str('\u03BE = ' + '{:05.3f}'.format(popt[0]) + - ' +- ' + '{:05.3f}'.format(pcov[0][0]**0.5) + - ' (' + '{:03.2f}'.format(pcov[0][0]**0.5 / abs(popt[0]) * 100) + '%)') - -fit_legend += str('\n\u03C3 = ' + '{:05.3f}'.format(popt[1]) + - ' +- ' + '{:05.3f}'.format(pcov[1][1]**0.5) + - ' (' + '{:03.2f}'.format(pcov[1][1]**0.5 / abs(popt[1]) * 100) + '%)') - -fit_legend += '\n\u03BC = ' + str(threshold) + ' (const)' - -fit_legend += '\ngoodness-of-fit: ' + '{:03.3f}'.format(chi2_test(hist_ot_norm, - n_bins=n_bins, - c=popt[0], - loc=threshold, - scale=popt[1], - norm=len(lat))) -print(fit_legend) - -ax[0][1].set_xscale('log') -##ax[0][1].set_yscale('log') -ax[0][1].set_xlabel('Return period', fontsize=8) -ax[0][1].set_ylabel('Return level [\u03BCs]', fontsize=6) -ax[0][1].grid(True, linestyle=":", which="both") - -y = np.linspace(200000, 5000000, 400) -ax[0][1].plot(y, - quantile(1 / y, - P=P, - c=popt[0], - loc=threshold, - scale=popt[1]), - 'r-', lw=1, color='blue', alpha=0.8) - -ax[0][1].plot(y, - quantile_conf_bound(1 / y, - P=P, - n=+1, - c=popt[0], - loc=threshold, - scale=popt[1], - err_P=err_P, - err_c= pcov[0][0]**0.5, - err_scale=pcov[1][1]**0.5), - 'r-', lw=1, color='green', alpha=0.8) - -ax[0][1].plot(y, - quantile_conf_bound(1 / y, - P=P, - n=-1, - c=popt[0], - loc=threshold, - scale=popt[1], - err_P=err_P, - err_c= pcov[0][0]**0.5, - err_scale=pcov[1][1]**0.5), - 'r-', lw=1, color='green', alpha=0.8) - -props = dict(boxstyle='round', color='black', alpha=0.05) - -ax[1][1].text(0.05, 0.85, - fit_legend, - fontsize=9, - verticalalignment='top', - bbox=props) - -plt.savefig('figfit-all-loaded.png') -#plt.show()
examples/page_faults.py deleted +0 -120
120 lines not on this page · raw message
examples/sched_wakeup.py +30 -40
132 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH v2 5/9 trace-cruncher: Add Makefile
In reply to Yordan Karadzhov (VMware)This will simplify the building procedure and will make it more intuitive.
--- Makefile | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 Makefile
Makefile new file +33 -0
@@ -0,0 +1,33 @@ +# +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2019 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> +# + +UID := $(shell id -u) + +CYAN := '\e[36m' +PURPLE := '\e[35m' +NC := '\e[0m' + +all: + @ echo ${CYAN}Buildinging trace-cruncher:${NC}; + python3 setup.py build + +clean: + rm -f src/npdatawrapper.c + rm -rf build + +install: + @ echo ${CYAN}Installing trace-cruncher:${NC}; + python3 setup.py install --record install_manifest.txt + +uninstall: + @ if [ $(UID) -ne 0 ]; then \ + echo ${PURPLE}Permission denied${NC} 1>&2; \ + else \ + echo ${CYAN}Uninstalling trace-cruncher:${NC}; \ + xargs rm -v < install_manifest.txt; \ + rm -rfv dist tracecruncher.egg-info; \ + rm -fv install_manifest.txt; \ + fi
Signature · 1 line
2.27.0
PATCH v2 6/9 trace-cruncher: Update README.md
In reply to Yordan Karadzhov (VMware)Building instruction are updated in order to properly describe the refactored version and the installation of all third-party dependencies.
--- README.md | 84 +++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 54 insertions(+), 30 deletions(-)
README.md +54 -30
@@ -4,68 +4,92 @@ ## Overview -The Trace-Cruncher project aims to provide an interface between the existing instrumentation for collection and visualization of tracing data from the Linux kernel and the broad and very well developed ecosystem of instruments for data analysis available in Python. The interface will be based on NumPy. +The Trace-Cruncher project aims to provide an interface between the existing instrumentation for collection and visualization of tracing data from the Linux kernel and the broad and very well developed ecosystem of instruments for data analysis available in Python. The interface is based on NumPy. -NumPy implements an efficient multi-dimensional container of generic data and uses strong typing in order to provide fast data processing in Python. The Trace-Cruncher will allow for sophisticated analysis of kernel tracing data via scripts, but it will also opens the door for exposing the kernel tracing data to the instruments provided by the scientific toolkit of Python like MatPlotLib, Stats, Scikit-Learn and even to the nowadays most popular frameworks for Machine Learning like TensorFlow and PyTorch. The Trace-Cruncher is strongly coupled to the KernelShark project and is build on top of the C API of libkshark. +NumPy implements an efficient multi-dimensional container of generic data and uses strong typing in order to provide fast data processing in Python. The Trace-Cruncher allows for sophisticated analysis of kernel tracing data via scripts, but it also opens the door for exposing the kernel tracing data to the instruments provided by the scientific toolkit of Python like MatPlotLib, Stats, Scikit-Learn and even to the nowadays most popular frameworks for Machine Learning like TensorFlow and PyTorch. The Trace-Cruncher is strongly coupled to the KernelShark project and is build on top of the C API of libkshark. ## Try it out ### Prerequisites Trace-Cruncher has the following external dependencies: - trace-cmd / KernelShark, Json-C, Cython, NumPy, MatPlotLib. + libtraceevent, libtracefs, KernelShark, Json-C, Cython, NumPy. -1.1 In order to install the packages on Ubuntu do the following: +1.1 In order to install all packages on Ubuntu do the following: - sudo apt-get install libjson-c-dev libpython3-dev cython3 -y + > sudo apt-get update - sudo apt-get install python3-numpy python3-matplotlib -y + > sudo apt-get install build-essential git cmake libjson-c-dev -y -1.2 In order to install the packages on Fedora, as root do the following: + > sudo apt-get install libpython3-dev cython3 python3-numpy python3-pip -y - dnf install json-c-devel python3-devel python3-Cython -y + > sudo pip3 install --system pkgconfig GitPython - dnf install python3-numpy python3-matplotlib -y +1.2 In order to install all packages on Fedora, as root do the following: -2. In order to get the proper version of KernelShark / trace-cmd do the -following: + > dnf install gcc gcc-c++ git cmake json-c-devel -y - git clone git://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git --branch=kernelshark-v1.1 + > dnf install python3-devel python3-Cython python3-numpy python3-pip -y -or download a tarball from here: -https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git/snapshot/trace-cmd-kernelshark-v1.1.tar.gz + > sudo pip3 install --system pkgconfig GitPython -### Build & Run -1. Patch trace-cmd / KernelShark: +2 In order to install all third party libraries do the following: + + > git clone https://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git/ + + > cd libtraceevent + + > make + + > sudo make install + + > cd .. + + + > git clone https://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git/ + + > cd libtracefs - cd path/to/trace-cmd/ + > make - git am ../path/to/trace-cruncher/0001-kernel-shark-Add-_DEVEL-build-flag.patch + > sudo make install - git am ../path/to/trace-cruncher/0002-kernel-shark-Add-reg_pid-plugin.patch + > cd .. -2. Install trace-cmd: - make + > git clone https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git - sudo make install_libs + > cd trace-cmd -3. Install KernelShark: + > make - cd kernel-shark/build + > sudo make install_libs - cmake -D_DEVEL=1 ../ + > cd .. - make - sudo make install + > git clone https://github.com/yordan-karadzhov/kernel-shark-v2.beta.git kernel-shark + + > cd kernel-shark/build + + > cmake .. + + > make + + > sudo make install + + > cd ../.. + +### Build & Run + +Installing trace-cruncher is very simple. After downloading the source code, you just have to run: -4. Build the NumPy API itself: + > cd trace-cruncher - cd path/to/trace-cruncher + > make - ./np_setup.py build_ext -i + > sudo make install ## Documentation
Signature · 1 line
2.27.0
PATCH v2 7/9 trace-cruncher: Remove all leftover files.
In reply to Yordan Karadzhov (VMware)This patch completes the refactoring of trace-cruncher into a Python module. All obsoleted source files are removed.
--- 0001-kernel-shark-Add-_DEVEL-build-flag.patch | 90 ----- 0002-kernel-shark-Add-reg_pid-plugin.patch | 231 ----------- clean.sh | 6 - examples/ksharksetup.py | 24 -- libkshark-py.c | 224 ----------- libkshark_wrapper.pyx | 361 ------------------ np_setup.py | 90 ----- 7 files changed, 1026 deletions(-) delete mode 100644 0001-kernel-shark-Add-_DEVEL-build-flag.patch delete mode 100644 0002-kernel-shark-Add-reg_pid-plugin.patch delete mode 100755 clean.sh delete mode 100644 examples/ksharksetup.py delete mode 100644 libkshark-py.c delete mode 100644 libkshark_wrapper.pyx delete mode 100755 np_setup.py
0001-kernel-shark-Add-_DEVEL-build-flag.patch deleted +0 -90
@@ -1,90 +0,0 @@ -From 6c9e3b3f29c8af4780bb46313c3af73fb5d852c7 Mon Sep 17 00:00:00 2001 -From: "Yordan Karadzhov (VMware)" <[email protected]> -Date: Fri, 20 Sep 2019 14:31:15 +0300 -Subject: [PATCH 1/2] kernel-shark: Add _DEVEL build flag - -KernelShark can be built with -D_DEVEL=1 as a command-line argument -for Cmake. In this case the headers of the libraries will be installed -as well and a symbolic link that points to the version of the library -being installed will be created. - -Signed-off-by: Yordan Karadzhov (VMware) <[email protected]> ---- - kernel-shark/README | 3 +++ - kernel-shark/src/CMakeLists.txt | 33 +++++++++++++++++++++++++++++++++ - 2 files changed, 36 insertions(+) - -diff --git a/kernel-shark/README b/kernel-shark/README -index 6c360bb..0f14212 100644 ---- a/kernel-shark/README -+++ b/kernel-shark/README -@@ -96,6 +96,9 @@ the dialog will derive the absolut path to the trace-cmd executable from - - If no build types is specified, the type will be "RelWithDebInfo". - -+2.1.4 In order to install a development version (including headers e.t.c) add -+-D_DEVEL=1 as a CMake Command-Line option. -+ - Examples: - - cmake -D_DOXYGEN_DOC=1 -D_INSTALL_PREFIX=/usr ../ -diff --git a/kernel-shark/src/CMakeLists.txt b/kernel-shark/src/CMakeLists.txt -index e20a030..305840b 100644 ---- a/kernel-shark/src/CMakeLists.txt -+++ b/kernel-shark/src/CMakeLists.txt -@@ -1,5 +1,13 @@ - message("\n src ...") - -+macro(install_symlink filepath sympath) -+ install(CODE "execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink ${filepath} ${sympath})") -+ install(CODE "LIST(APPEND CMAKE_INSTALL_MANIFEST_FILES ${sympath})") -+ install(CODE "message(\"-- Created symlink: ${sympath} -> ${filepath}\")") -+endmacro(install_symlink) -+ -+set(KS_INCLUDS_DESTINATION "${_INSTALL_PREFIX}/include/${KS_APP_NAME}") -+ - message(STATUS "libkshark") - add_library(kshark SHARED libkshark.c - libkshark-model.c -@@ -16,6 +24,19 @@ set_target_properties(kshark PROPERTIES SUFFIX ".so.${KS_VERSION_STRING}") - - install(TARGETS kshark LIBRARY DESTINATION ${_INSTALL_PREFIX}/lib/${KS_APP_NAME}) - -+if (_DEVEL) -+ -+ install_symlink("libkshark.so.${KS_VERSION_STRING}" -+ "${_INSTALL_PREFIX}/lib/${KS_APP_NAME}/libkshark.so") -+ -+ install(FILES "${KS_DIR}/src/libkshark.h" -+ "${KS_DIR}/src/libkshark-plugin.h" -+ "${KS_DIR}/src/libkshark-model.h" -+ DESTINATION ${KS_INCLUDS_DESTINATION} -+ COMPONENT devel) -+ -+endif (_DEVEL) -+ - if (OPENGL_FOUND AND GLUT_FOUND) - - message(STATUS "libkshark-plot") -@@ -30,6 +51,18 @@ if (OPENGL_FOUND AND GLUT_FOUND) - - install(TARGETS kshark-plot LIBRARY DESTINATION ${_INSTALL_PREFIX}/lib/${KS_APP_NAME}) - -+ if (_DEVEL) -+ -+ install_symlink("libkshark-plot.so.${KS_VERSION_STRING}" -+ "${_INSTALL_PREFIX}/lib/${KS_APP_NAME}/libkshark-plot.so") -+ -+ install(FILES "${KS_DIR}/src/KsPlotTools.hpp" -+ "${KS_DIR}/src/libkshark-plot.h" -+ DESTINATION ${KS_INCLUDS_DESTINATION} -+ COMPONENT devel) -+ -+ endif (_DEVEL) -+ - endif (OPENGL_FOUND AND GLUT_FOUND) - - if (Qt5Widgets_FOUND AND Qt5Network_FOUND) --- -2.20.1 -
0002-kernel-shark-Add-reg_pid-plugin.patch deleted +0 -231
231 lines not on this page · raw message
clean.sh deleted +0 -6
@@ -1,6 +0,0 @@ -#!/bin/bash - -rm libkshark_wrapper.c -rm ksharkpy.cpython-3*.so -rm -rf build/ -rm -rf examples/__pycache__/
examples/ksharksetup.py deleted +0 -24
@@ -1,24 +0,0 @@ -#!/usr/bin/env python3 - -""" -SPDX-License-Identifier: LGPL-2.1 - -Copyright 2019 VMware Inc, Yordan Karadzhov <[email protected]> -""" - -import os -import sys - -def setup(): - os.chdir(os.path.dirname(__file__)) - path = os.getcwd() + '/..' - sys.path.append(path) - - if 'LD_LIBRARY_PATH' not in os.environ: - os.environ['LD_LIBRARY_PATH'] = '/usr/local/lib/kernelshark:/usr/local/lib/traceevent:/usr/local/lib/trace-cmd' - try: - os.execv(sys.argv[0], sys.argv) - except e: - print('Failed re-exec:', e) - sys.exit(1) -
libkshark-py.c deleted +0 -224
224 lines not on this page · raw message
libkshark_wrapper.pyx deleted +0 -361
361 lines not on this page · raw message
np_setup.py deleted +0 -90
90 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH v2 8/9 trace-cruncher: Add testing
In reply to Yordan Karadzhov (VMware)Add basic infrastructure for CI testing.
--- tests/0_get_data/__init__.py | 0 tests/0_get_data/test_get_data.py | 26 + tests/1_unit/__init__.py | 0 tests/1_unit/test_01_ftracepy_unit.py | 495 ++++++++++++++++++ tests/1_unit/test_02_datawrapper_unit.py | 41 ++ tests/1_unit/test_03_ksharkpy_unit.py | 72 +++ tests/2_integration/__init__.py | 0 .../test_01_ftracepy_integration.py | 113 ++++ .../test_03_ksharkpy_integration.py | 25 + tests/__init__.py | 0 10 files changed, 772 insertions(+) create mode 100644 tests/0_get_data/__init__.py create mode 100755 tests/0_get_data/test_get_data.py create mode 100644 tests/1_unit/__init__.py create mode 100644 tests/1_unit/test_01_ftracepy_unit.py create mode 100755 tests/1_unit/test_02_datawrapper_unit.py create mode 100755 tests/1_unit/test_03_ksharkpy_unit.py create mode 100644 tests/2_integration/__init__.py create mode 100755 tests/2_integration/test_01_ftracepy_integration.py create mode 100755 tests/2_integration/test_03_ksharkpy_integration.py create mode 100644 tests/__init__.py
tests/0_get_data/__init__.py new file +0 -0
tests/0_get_data/test_get_data.py new file +26 -0
@@ -0,0 +1,26 @@ +#!/usr/bin/env python3 + +""" +SPDX-License-Identifier: LGPL-2.1 + +Copyright (C) 2021 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> +""" + +import os +import sys +import shutil +import unittest +import git + +class GetTestData(unittest.TestCase): + def test_get_data(self): + data_dir = 'testdata' + if os.path.exists(data_dir) and os.path.isdir(data_dir): + shutil.rmtree(data_dir) + + github_repo = 'https://github.com/yordan-karadzhov/kernel-shark_testdata.git' + repo = git.Repo.clone_from(github_repo, data_dir) + + +if __name__ == '__main__': + unittest.main()
tests/1_unit/__init__.py new file +0 -0
tests/1_unit/test_01_ftracepy_unit.py new file +495 -0
495 lines not on this page · raw message
tests/1_unit/test_02_datawrapper_unit.py new file +41 -0
41 lines not on this page · raw message
tests/1_unit/test_03_ksharkpy_unit.py new file +72 -0
72 lines not on this page · raw message
tests/2_integration/__init__.py new file +0 -0
tests/2_integration/test_01_ftracepy_integration.py new file +113 -0
113 lines not on this page · raw message
tests/2_integration/test_03_ksharkpy_integration.py new file +25 -0
@@ -0,0 +1,25 @@ +#!/usr/bin/env python3 + +""" +SPDX-License-Identifier: LGPL-2.1 + +Copyright (C) 2021 VMware Inc, Yordan Karadzhov (VMware) <[email protected]> +""" + +import os +import sys +import shutil +import unittest +import tracecruncher.ks_utils as tc + +class GetTestData(unittest.TestCase): + def test_open_and_read(self): + f = tc.open_file(file_name='testdata/trace_test1.dat') + data = f.load(pid_data=False) + tasks = f.get_tasks() + self.assertEqual(len(tasks), 29) + self.assertEqual(tasks['zoom'], [28121, 28137, 28141, 28199, 28201, 205666]) + + +if __name__ == '__main__': + unittest.main()
tests/__init__.py new file +0 -0
Signature · 1 line
2.27.0
PATCH v2 9/9 trace-cruncher: Add github workflow for CI testing
In reply to Yordan Karadzhov (VMware)The CI workflow will run once a week, or on any push to branches "master" and "yordan_devel".
--- .github/workflows/main.yml | 58 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 .github/workflows/main.yml
.github/workflows/main.yml new file +58 -0
@@ -0,0 +1,58 @@ +name: trace-cruncher CI + +on: + push: + branches: [master, yordan_devel] + schedule: + - cron: '0 15 * * THU' + +jobs: + build: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + + - name: Install Dependencies + shell: bash + run: | + sudo apt-get update + sudo apt-get install build-essential git cmake libjson-c-dev -y + sudo apt-get install libpython3-dev cython3 python3-numpy -y + sudo apt install python3-pip + sudo pip3 install --system pkgconfig GitPython + git clone https://git.kernel.org/pub/scm/libs/libtrace/libtraceevent.git/ + cd libtraceevent + make + sudo make install + cd .. + git clone https://git.kernel.org/pub/scm/libs/libtrace/libtracefs.git/ + cd libtracefs + make + sudo make install + cd .. + git clone https://git.kernel.org/pub/scm/utils/trace-cmd/trace-cmd.git + cd trace-cmd + make + sudo make install_libs + cd .. + git clone https://github.com/yordan-karadzhov/kernel-shark-v2.beta.git kernel-shark + cd kernel-shark/build + cmake .. + make + sudo make install + cd ../.. + + - name: Build + working-directory: ${{runner.workspace}}/trace-cruncher + shell: bash + # Build and install. + run: | + make + sudo make install + + - name: Test + working-directory: ${{runner.workspace}}/trace-cruncher/tests + shell: bash + # Execute tests defined by the CMake configuration. + run: sudo python3 -m unittest discover .
Signature · 1 line
2.27.0