GSoC 2020 - TCG Continuous Benchmarking scripts and tools
cover GSoC 2020 - TCG Continuous Benchmarking scripts and tools
Greetings,
This series includes all of the scripts, tools, and benchmarks developed in the TCG Continuous Benchmarking GSoC project for 2020.
The series includes one patch for updating the MAINTAINERS file and eight patches each with a separate script or tool.
All scripts and tools were thoroughly introduced, explained, and utilized in the project weekly reports which were posted here on the list each Monday at 12:30PM for the last three months.
Reports:
Report 1 - Measuring Basic Performance Metrics of QEMU:
https://www.mail-archive.com/[email protected]/msg714853.html
Introduced tools:topN_perf.py and topN_callgrind.py
Report 2 - Dissecting QEMU Into Three Main Parts:
https://www.mail-archive.com/[email protected]/msg717592.html
Introduced tools:dissect.py
Report 3 - QEMU 5.0 and 5.1-pre-soft-freeze Dissect Comparison:
https://www.mail-archive.com/[email protected]/msg720321.html
Report 4 - Listing QEMU Helpers and Function Callees:
https://www.mail-archive.com/[email protected]/msg722559.html
Introduced tools:list_fn_callees.py and list_helpers.py
Report 5 - Finding Commits Affecting QEMU Performance:
https://www.mail-archive.com/[email protected]/msg724080.html
Introduced tools:bisect.py
Report 6 - Performance Comparison of Two QEMU Builds:
https://www.mail-archive.com/[email protected]/msg725700.html
Report 7 - Measuring QEMU Emulation Efficiency:
https://www.mail-archive.com/[email protected]/msg727147.html
Report 8 - QEMU Nightly Performance Tests:
https://www.mail-archive.com/[email protected]/msg730444.html
Introduced tools:
nightly-tests/
Report 9 - Measuring QEMU Performance in System Mode:
https://www.mail-archive.com/[email protected]/msg732729.html
Introduced tools:topN_system.py
Report 10 - Measuring QEMU Performance in System Mode - Part Two:
https://www.mail-archive.com/[email protected]/msg734485.html
Best regards,
Ahmed Karaman
Ahmed Karaman (9):
scripts/performance: Refactor topN_perf.py scripts/performance: Refactor topN_callgrind.py scripts/performance: Refactor dissect.py scripts/performance: Add list_fn_callees.py script scripts/performance: Add list_helpers.py script scripts/performance: Add bisect.py script tests/performance: Add nightly tests
MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests'
subsection
scripts/performance: Add topN_system.py script
MAINTAINERS | 32 +- scripts/performance/bisect.py | 425 ++++++++ scripts/performance/dissect.py | 123 +-- scripts/performance/list_fn_callees.py | 245 +++++ scripts/performance/list_helpers.py | 221 +++++ scripts/performance/topN_callgrind.py | 169 ++-- scripts/performance/topN_perf.py | 174 ++-- scripts/performance/topN_system.py | 158 +++ tests/performance/nightly-tests/README.md | 243 +++++ .../source/dijkstra_double/dijkstra_double.c | 194 ++++ .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ .../source/matmult_double/matmult_double.c | 123 +++ .../source/matmult_int32/matmult_int32.c | 121 +++ .../source/qsort_double/qsort_double.c | 104 ++ .../source/qsort_int32/qsort_int32.c | 103 ++ .../source/qsort_string/qsort_string.c | 122 +++ .../source/search_string/search_string.c | 110 +++ .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ .../scripts/run_nightly_tests.py | 135 +++ .../nightly-tests/scripts/send_email.py | 56 ++ 20 files changed, 3744 insertions(+), 226 deletions(-) create mode 100755 scripts/performance/bisect.py create mode 100755 scripts/performance/list_fn_callees.py create mode 100755 scripts/performance/list_helpers.py create mode 100755 scripts/performance/topN_system.py create mode 100644 tests/performance/nightly-tests/README.md create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py create mode 100644 tests/performance/nightly-tests/scripts/send_email.py
Signature · 1 line
2.17.1
PATCH 1/9 scripts/performance: Refactor topN_perf.py
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.
--- scripts/performance/topN_perf.py | 174 +++++++++++++++---------------- 1 file changed, 87 insertions(+), 87 deletions(-)
scripts/performance/topN_perf.py +87 -87
@@ -1,72 +1,77 @@ #!/usr/bin/env python3 -# Print the top N most executed functions in QEMU using perf. -# Syntax: -# topN_perf.py [-h] [-n] <number of displayed top functions> -- \ -# <qemu executable> [<qemu executable options>] \ -# <target executable> [<target execurable options>] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman <[email protected]> -# Copyright (C) 2020 Aleksandar Markovic <[email protected]> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <https://www.gnu.org/licenses/>. +""" +Print the top N most executed functions in QEMU using perf. + +Syntax: +topN_perf.py [-h] [-n <number of displayed top functions>] -- \ + <qemu executable> [<qemu executable options>] \ + <target executable> [<target execurable options>] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_perf.py -n 20 -- qemu-arm coulomb_double-arm + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" import argparse import os import subprocess import sys +import tempfile # Parse the command line arguments -parser = argparse.ArgumentParser( - usage='topN_perf.py [-h] [-n] <number of displayed top functions > -- ' +PARSER = argparse.ArgumentParser( + usage='topN_perf.py [-h] [-n <number of displayed top functions>] -- ' '<qemu executable> [<qemu executable options>] ' '<target executable> [<target executable options>]') -parser.add_argument('-n', dest='top', type=int, default=25, +PARSER.add_argument('-n', dest='top', type=int, default=25, help='Specify the number of top functions to print.') -parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) +PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) -args = parser.parse_args() +ARGS = PARSER.parse_args() # Extract the needed variables from the args -command = args.command -top = args.top +COMMAND = ARGS.command +TOP = ARGS.top # Insure that perf is installed -check_perf_presence = subprocess.run(["which", "perf"], - stdout=subprocess.DEVNULL) -if check_perf_presence.returncode: +CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"], + stdout=subprocess.DEVNULL, + check=False) +if CHECK_PERF_PRESENCE.returncode: sys.exit("Please install perf before running the script!") # Insure user has previllage to run perf -check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"], +CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"], stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL) -if check_perf_executability.returncode: - sys.exit( -""" + stderr=subprocess.DEVNULL, + check=False) +if CHECK_PERF_EXECUTABILITY.returncode: + sys.exit(""" Error: You may not have permission to collect stats.
@@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conf too, e.g.: kernel.perf_event_paranoid = -1 * Alternatively, you can run this script under sudo privileges. -""" -) - -# Run perf record -perf_record = subprocess.run((["perf", "record", "--output=/tmp/perf.data"] + - command), - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) -if perf_record.returncode: - os.unlink('/tmp/perf.data') - sys.exit(perf_record.stderr.decode("utf-8")) - -# Save perf report output to /tmp/perf_report.out -with open("/tmp/perf_report.out", "w") as output: - perf_report = subprocess.run( - ["perf", "report", "--input=/tmp/perf.data", "--stdio"], - stdout=output, - stderr=subprocess.PIPE) - if perf_report.returncode: - os.unlink('/tmp/perf.data') - output.close() - os.unlink('/tmp/perf_report.out') - sys.exit(perf_report.stderr.decode("utf-8")) - -# Read the reported data to functions[] -functions = [] -with open("/tmp/perf_report.out", "r") as data: - # Only read lines that are not comments (comments start with #) - # Only read lines that are not empty - functions = [line for line in data.readlines() if line and line[0] - != '#' and line[0] != "\n"] - -# Limit the number of top functions to "top" -number_of_top_functions = top if len(functions) > top else len(functions) - -# Store the data of the top functions in top_functions[] -top_functions = functions[:number_of_top_functions] +""") + +# Run perf and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + RECORD_PATH = os.path.join(tmpdir, "record.data") + REPORT_PATH = os.path.join(tmpdir, "report.txt") + + PERF_RECORD = subprocess.run((["perf", "record", "--output="+RECORD_PATH] + + COMMAND), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if PERF_RECORD.returncode: + sys.exit(PERF_RECORD.stderr.decode("utf-8")) + + with open(REPORT_PATH, "w") as output: + PERF_REPORT = subprocess.run( + ["perf", "report", "--input="+RECORD_PATH, "--stdio"], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if PERF_REPORT.returncode: + sys.exit(PERF_REPORT.stderr.decode("utf-8")) + + # Save the reported data to FUNCTIONS[] + with open(REPORT_PATH, "r") as data: + # Only read lines that are not comments (comments start with #) + # Only read lines that are not empty + FUNCTIONS = [line for line in data.readlines() if line and + line[0] != '#' and line[0] != "\n"] + +# Limit the number of top functions to "TOP" +NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS) + +# Store the data of the top functions in TOP_FUNCTIONS[] +TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS] # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
@@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', '-' * 25)) # Print top N functions -for (index, function) in enumerate(top_functions, start=1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=1): function_data = function.split() function_percentage = function_data[0] function_name = function_data[-1]
@@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions, start=1): function_percentage, function_name, function_invoker)) - -# Remove intermediate files -os.unlink('/tmp/perf.data') -os.unlink('/tmp/perf_report.out')
Signature · 1 line
2.17.1
Re: [PATCH 1/9] scripts/performance: Refactor topN_perf.py
Quoted 4 lines · Ahmed Karaman
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.
Reply
Reply all
Forward
View Gmail in: *Mobile* | Older version
<https://mail.google.com/mail/mu/mp/966/#> | Desktop
<https://mail.google.com/mail/mu/mp/966/#>
© 2020 Google
Quoted 243 lines
Signed-off-by Ahmed Karaman <[email protected]>--- scripts/performance/topN_perf.py | 174 +++++++++++++++---------------- 1 file changed, 87 insertions(+), 87 deletions(-)scripts/performance/topN_ renamed from scripts/performance/topN_perf.py +0
-0
perf.py
index07be195fc8..56b100da87100755scripts/performance/topN_perf.py +0
-17@@ -1,72 +1,77 @@ #!/usr/bin/env python3 -# Print the top N most executed functions in QEMU using perf. -# Syntax: -# topN_perf.py [-h] [-n] <number of displayed top functions> -- \ -# <qemu executable> [<qemu executable options>] \ -# <target executable> [<target execurable options>] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman <[email protected]> -# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
scripts/performance/topN_perf.py +37
-18-# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <https://www.gnu.org/licenses/>. +""" +Print the top N most executed functions in QEMU using perf. + +Syntax: +topN_perf.py [-h] [-n <number of displayed top functions>] -- \ + <qemu executable> [<qemu executable options>] \ + <target executable> [<target execurable options>] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_perf.py -n 20 -- qemu-arm coulomb_double-arm + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" import argparse import os import subprocess import sys +import tempfile # Parse the command line arguments -parser = argparse.ArgumentParser( - usage='topN_perf.py [-h] [-n] <number of displayed top functions > -- ' +PARSER = argparse.ArgumentParser( + usage='topN_perf.py [-h] [-n <number of displayed top functions>] -- ' '<qemu executable> [<qemu executable options>] ' '<target executable> [<target executable options>]') -parser.add_argument('-n', dest='top', type=int, default=25, +PARSER.add_argument('-n', dest='top', type=int, default=25, help='Specify the number of top functions to print.') -parser.add_argument('command', type=str, nargs='+',help=argparse.SUPPRESS) +PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
scripts/performance/topN_perf.py +12
-11-args = parser.parse_args() +ARGS = PARSER.parse_args() # Extract the needed variables from the args -command = args.command -top = args.top +COMMAND = ARGS.command +TOP = ARGS.top # Insure that perf is installed -check_perf_presence = subprocess.run(["which", "perf"], - stdout=subprocess.DEVNULL) -if check_perf_presence.returncode: +CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"], + stdout=subprocess.DEVNULL, + check=False) +if CHECK_PERF_PRESENCE.returncode: sys.exit("Please install perf before running the script!") # Insure user has previllage to run perf -check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"], +CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"], stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL) -if check_perf_executability.returncode: - sys.exit( -""" + stderr=subprocess.DEVNULL, + check=False) +if CHECK_PERF_EXECUTABILITY.returncode: + sys.exit(""" Error: You may not have permission to collect stats.@@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conftoo, e.g.:
kernel.perf_event_paranoid = -1* Alternatively, you can run this script under sudo privileges.
-"""scripts/performance/topN_perf.py +0
-4-) - -# Run perf record -perf_record = subprocess.run((["perf", "record","--output=/tmp/perf.data"] +
scripts/performance/topN_perf.py +8
-32- command), - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) -if perf_record.returncode: - os.unlink('/tmp/perf.data') - sys.exit(perf_record.stderr.decode("utf-8")) - -# Save perf report output to /tmp/perf_report.out -with open("/tmp/perf_report.out", "w") as output: - perf_report = subprocess.run( - ["perf", "report", "--input=/tmp/perf.data", "--stdio"], - stdout=output, - stderr=subprocess.PIPE) - if perf_report.returncode: - os.unlink('/tmp/perf.data') - output.close() - os.unlink('/tmp/perf_report.out') - sys.exit(perf_report.stderr.decode("utf-8")) - -# Read the reported data to functions[] -functions = [] -with open("/tmp/perf_report.out", "r") as data: - # Only read lines that are not comments (comments start with #) - # Only read lines that are not empty - functions = [line for line in data.readlines() if line and line[0] - != '#' and line[0] != "\n"] - -# Limit the number of top functions to "top" -number_of_top_functions = top if len(functions) > top else len(functions) - -# Store the data of the top functions in top_functions[] -top_functions = functions[:number_of_top_functions] +""") + +# Run perf and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + RECORD_PATH = os.path.join(tmpdir, "record.data") + REPORT_PATH = os.path.join(tmpdir, "report.txt") + + PERF_RECORD = subprocess.run((["perf", "record","--output="+RECORD_PATH] +
scripts/performance/topN_perf.py +29
-1+ COMMAND), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if PERF_RECORD.returncode: + sys.exit(PERF_RECORD.stderr.decode("utf-8")) + + with open(REPORT_PATH, "w") as output: + PERF_REPORT = subprocess.run( + ["perf", "report", "--input="+RECORD_PATH, "--stdio"], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if PERF_REPORT.returncode: + sys.exit(PERF_REPORT.stderr.decode("utf-8")) + + # Save the reported data to FUNCTIONS[] + with open(REPORT_PATH, "r") as data: + # Only read lines that are not comments (comments start with #) + # Only read lines that are not empty + FUNCTIONS = [line for line in data.readlines() if line and + line[0] != '#' and line[0] != "\n"] + +# Limit the number of top functions to "TOP" +NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS) + +# Store the data of the top functions in TOP_FUNCTIONS[] +TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS] # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',@@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', '-' * 25)) # Print top N functions -for (index, function) in enumerate(top_functions, start=1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=1): function_data = function.split() function_percentage = function_data[0]
function_name= function_data[-1]scripts/performance/topN_perf.py +0
-0@@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions,start=1):
function_percentage,function_name,function_invoker))scripts/performance/topN_perf.py +0
-5- -# Remove intermediate files -os.unlink('/tmp/perf.data') -os.unlink('/tmp/perf_report.out') --2.17.1
Re: [PATCH 1/9] scripts/performance: Refactor topN_perf.py
In reply to Ahmed KaramanQuoted 5 lines · Ahmed Karaman
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.Signed-off-by Ahmed Karaman <[email protected]>---
Hello, folks.
This series seems forgotten. Can some of you perhaps take a look, review, and possibly integrate some patches in a pull request?
Ahmed invested quite of time to improve the functionality and quality of scripts, and they are truly useful for developers interested in performance measurement.
Thanks,
Aleksandar
Quoted 242 lines
scripts/performance/topN_perf.py | 174 +++++++++++++++---------------- 1 file changed, 87 insertions(+), 87 deletions(-)scripts/performance/topN_ renamed from scripts/performance/topN_perf.py +0
-0
perf.py
index07be195fc8..56b100da87100755scripts/performance/topN_perf.py +0
-17@@ -1,72 +1,77 @@ #!/usr/bin/env python3 -# Print the top N most executed functions in QEMU using perf. -# Syntax: -# topN_perf.py [-h] [-n] <number of displayed top functions> -- \ -# <qemu executable> [<qemu executable options>] \ -# <target executable> [<target execurable options>] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman <[email protected]> -# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
scripts/performance/topN_perf.py +37
-18-# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <https://www.gnu.org/licenses/>. +""" +Print the top N most executed functions in QEMU using perf. + +Syntax: +topN_perf.py [-h] [-n <number of displayed top functions>] -- \ + <qemu executable> [<qemu executable options>] \ + <target executable> [<target execurable options>] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_perf.py -n 20 -- qemu-arm coulomb_double-arm + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" import argparse import os import subprocess import sys +import tempfile # Parse the command line arguments -parser = argparse.ArgumentParser( - usage='topN_perf.py [-h] [-n] <number of displayed top functions > -- ' +PARSER = argparse.ArgumentParser( + usage='topN_perf.py [-h] [-n <number of displayed top functions>] -- ' '<qemu executable> [<qemu executable options>] ' '<target executable> [<target executable options>]') -parser.add_argument('-n', dest='top', type=int, default=25, +PARSER.add_argument('-n', dest='top', type=int, default=25, help='Specify the number of top functions to print.') -parser.add_argument('command', type=str, nargs='+',help=argparse.SUPPRESS) +PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
scripts/performance/topN_perf.py +12
-11-args = parser.parse_args() +ARGS = PARSER.parse_args() # Extract the needed variables from the args -command = args.command -top = args.top +COMMAND = ARGS.command +TOP = ARGS.top # Insure that perf is installed -check_perf_presence = subprocess.run(["which", "perf"], - stdout=subprocess.DEVNULL) -if check_perf_presence.returncode: +CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"], + stdout=subprocess.DEVNULL, + check=False) +if CHECK_PERF_PRESENCE.returncode: sys.exit("Please install perf before running the script!") # Insure user has previllage to run perf -check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"], +CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"], stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL) -if check_perf_executability.returncode: - sys.exit( -""" + stderr=subprocess.DEVNULL, + check=False) +if CHECK_PERF_EXECUTABILITY.returncode: + sys.exit(""" Error: You may not have permission to collect stats.@@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conftoo, e.g.:
kernel.perf_event_paranoid = -1* Alternatively, you can run this script under sudo privileges.
-"""scripts/performance/topN_perf.py +0
-4-) - -# Run perf record -perf_record = subprocess.run((["perf", "record","--output=/tmp/perf.data"] +
scripts/performance/topN_perf.py +8
-32- command), - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) -if perf_record.returncode: - os.unlink('/tmp/perf.data') - sys.exit(perf_record.stderr.decode("utf-8")) - -# Save perf report output to /tmp/perf_report.out -with open("/tmp/perf_report.out", "w") as output: - perf_report = subprocess.run( - ["perf", "report", "--input=/tmp/perf.data", "--stdio"], - stdout=output, - stderr=subprocess.PIPE) - if perf_report.returncode: - os.unlink('/tmp/perf.data') - output.close() - os.unlink('/tmp/perf_report.out') - sys.exit(perf_report.stderr.decode("utf-8")) - -# Read the reported data to functions[] -functions = [] -with open("/tmp/perf_report.out", "r") as data: - # Only read lines that are not comments (comments start with #) - # Only read lines that are not empty - functions = [line for line in data.readlines() if line and line[0] - != '#' and line[0] != "\n"] - -# Limit the number of top functions to "top" -number_of_top_functions = top if len(functions) > top else len(functions) - -# Store the data of the top functions in top_functions[] -top_functions = functions[:number_of_top_functions] +""") + +# Run perf and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + RECORD_PATH = os.path.join(tmpdir, "record.data") + REPORT_PATH = os.path.join(tmpdir, "report.txt") + + PERF_RECORD = subprocess.run((["perf", "record","--output="+RECORD_PATH] +
scripts/performance/topN_perf.py +29
-1+ COMMAND), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if PERF_RECORD.returncode: + sys.exit(PERF_RECORD.stderr.decode("utf-8")) + + with open(REPORT_PATH, "w") as output: + PERF_REPORT = subprocess.run( + ["perf", "report", "--input="+RECORD_PATH, "--stdio"], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if PERF_REPORT.returncode: + sys.exit(PERF_REPORT.stderr.decode("utf-8")) + + # Save the reported data to FUNCTIONS[] + with open(REPORT_PATH, "r") as data: + # Only read lines that are not comments (comments start with #) + # Only read lines that are not empty + FUNCTIONS = [line for line in data.readlines() if line and + line[0] != '#' and line[0] != "\n"] + +# Limit the number of top functions to "TOP" +NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS) + +# Store the data of the top functions in TOP_FUNCTIONS[] +TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS] # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',@@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', '-' * 25)) # Print top N functions -for (index, function) in enumerate(top_functions, start=1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=1): function_data = function.split() function_percentage = function_data[0]
function_name= function_data[-1]scripts/performance/topN_perf.py +0
-0@@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions,start=1):
function_percentage,function_name,function_invoker))scripts/performance/topN_perf.py +0
-5- -# Remove intermediate files -os.unlink('/tmp/perf.data') -os.unlink('/tmp/perf_report.out') --2.17.1
Re: Bottleneck problem to merge Python patches
Hi Aleksandar,
(extending the Cc list to broader audience)
Quoted 9 lines · Aleksandar Markovic
On Friday, August 28, 2020, Ahmed Karaman <[email protected] <mailto:[email protected]>> wrote:
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.---
Quoted 8 lines
Hello, folks.
This series seems forgotten. Can some of you perhaps take a look, review, and possibly integrate some patches in a pull request?
Ahmed invested quite of time to improve the functionality and quality of scripts, and they are truly useful for developers interested in performance measurement.
The Python patches are indeed being merged slowly, but are not forgotten :) Eduardo sent just a pull request *yesterday* for patches he had queued *before* QEMU 5.1, that is more than 1 year ago!
https://www.mail-archive.com/[email protected]/msg742228.html
I hope he will be able to process the other Python patches sent the last 12 months. He raised the problem few month ago saying he was overwhelmed and was looking for help from the community. Cleber helped a pair of times, I helped once, but then nobody popped up volunteering.
I agree this is a community problem, nobody wants to become the de-facto Python maintainer. Current maintainers are already busy solving problem with low-level languages such C.
As a project, QEMU is evolving, using more and more Python, switched to Meson, we might have Rust code too. Learning that for current maintainers takes time. I guess we lack new contributors/maintainers with interest in Python & QEMU.
This is my simple/rough analysis, as John had the same problem 2/3 months ago, his patches were on the list unreviewed for various weeks. Same problem with Avocado patches, Lukas sent a series a bit before Ahmed and it is still unreviewed:
https://www.mail-archive.com/[email protected]/msg737272.html
Alex Bennée recently said:
"review time is the currency of qemu-devel ;-)"
We might be failing at explaining new contributors the importance of peer-review, as this helps a lot maintainers. It is described in the wiki but maybe new contributors don't read it, we don't remember them to:
https://wiki.qemu.org/Contribute/SubmitAPatch#Participating_in_Code_Review
and:
https://wiki.qemu.org/Contribute/SubmitAPatch#Return_the_favor
My 2 cents...
Regards,
Phil.
Quoted 3 lines
Thanks,
Aleksandar
Re: Bottleneck problem to merge Python patches
Quoted 23 lines · Philippe Mathieu-Daudé
Hi Aleksandar,
(extending the Cc list to broader audience)
Quoted 17 lines · Aleksandar Markovic
On Friday, August 28, 2020, Ahmed Karaman <[email protected] <mailto:[email protected]>> wrote:
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.---
Hello, folks.
This series seems forgotten. Can some of you perhaps take a look, review, and possibly integrate some patches in a pull request?
Ahmed invested quite of time to improve the functionality and quality of scripts, and they are truly useful for developers interested in performance measurement.
Hi, I will add it to my queue. I intended to take over Python maintenance but I have been busy refactoring the QAPI python code and haven't been reading my mail as regularly.
Quoted 12 lines
The Python patches are indeed being merged slowly, but are not forgotten :) Eduardo sent just a pull request *yesterday* for patches he had queued *before* QEMU 5.1, that is more than 1 year ago!
https://www.mail-archive.com/[email protected]/msg742228.htmlI hope he will be able to process the other Python patches sent the last 12 months. He raised the problem few month ago saying he was overwhelmed and was looking for help from the community. Cleber helped a pair of times, I helped once, but then nobody popped up volunteering.
Quoted 8 lines
I agree this is a community problem, nobody wants to become the de-facto Python maintainer. Current maintainers are already busy solving problem with low-level languages such C.
As a project, QEMU is evolving, using more and more Python, switched to Meson, we might have Rust code too. Learning that for current maintainers takes time. I guess we lack new contributors/maintainers with interest in Python & QEMU.
I'm volunteering, since I am doing so much work in Python. I could use a dedicated reviewer to help me, however. I prefer a maintainer policy where all patches get at least ONE review by someone other than the primary author.
In the case that I am writing so much Python, I still need a co-maintainer to help review *my* patches.
Quoted 28 lines
This is my simple/rough analysis, as John had the same problem 2/3 months ago, his patches were on the list unreviewed for various weeks. Same problem with Avocado patches, Lukas sent a series a bit before Ahmed and it is still unreviewed:
https://www.mail-archive.com/[email protected]/msg737272.htmlAlex Bennée recently said:
"review time is the currency of qemu-devel ;-)"
We might be failing at explaining new contributors the importance of peer-review, as this helps a lot maintainers. It is described in the wiki but maybe new contributors don't read it, we don't remember them to:
https://wiki.qemu.org/Contribute/SubmitAPatch#Participating_in_Code_Review
and:
https://wiki.qemu.org/Contribute/SubmitAPatch#Return_the_favorMy 2 cents...
Regards,
Phil.
Quoted 3 lines
Thanks,
Aleksandar
Re: Bottleneck problem to merge Python patches
In reply to Philippe Mathieu-DaudéQuoted 29 lines · Philippe Mathieu-Daudé
Hi Aleksandar,
(extending the Cc list to broader audience)
Quoted 9 lines · Aleksandar Markovic
On Friday, August 28, 2020, Ahmed Karaman <[email protected] <mailto:[email protected]>> wrote:
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.---
Quoted 8 lines
Hello, folks.
This series seems forgotten. Can some of you perhaps take a look, review, and possibly integrate some patches in a pull request?
Ahmed invested quite of time to improve the functionality and quality of scripts, and they are truly useful for developers interested in performance measurement.
The Python patches are indeed being merged slowly, but are not forgotten :) Eduardo sent just a pull request *yesterday* for patches he had queued *before* QEMU 5.1, that is more than 1 year ago!
https://www.mail-archive.com/[email protected]/msg742228.html
Quoted 5 lines
I hope he will be able to process the other Python patches sent the last 12 months. He raised the problem few month ago saying he was overwhelmed and was looking for help from the community. Cleber helped a pair of times, I helped once, but then nobody popped up volunteering.
The patches were from July this year. But, yeah, I'm not being able to keep up with the maintainer role for Python code, and plan to change my M: line to R: in the "Python scripts" section.
Quoted 8 lines
I agree this is a community problem, nobody wants to become the de-facto Python maintainer. Current maintainers are already busy solving problem with low-level languages such C.
As a project, QEMU is evolving, using more and more Python, switched to Meson, we might have Rust code too. Learning that for current maintainers takes time. I guess we lack new contributors/maintainers with interest in Python & QEMU.
I believe the solution here is to not have a single owner for all Python code, but owners for specific areas. We already have 36
F: entries in MAINTAINERS for files inside scripts/, and each of those maintainers can send pull requests containing Python code in the areas they maintain. Including Ahmed, who is already listed as the maintainer of scripts/performance/.
Quoted 29 lines
This is my simple/rough analysis, as John had the same problem 2/3 months ago, his patches were on the list unreviewed for various weeks. Same problem with Avocado patches, Lukas sent a series a bit before Ahmed and it is still unreviewed:
https://www.mail-archive.com/[email protected]/msg737272.htmlAlex Bennée recently said:
"review time is the currency of qemu-devel ;-)"
We might be failing at explaining new contributors the importance of peer-review, as this helps a lot maintainers. It is described in the wiki but maybe new contributors don't read it, we don't remember them to:
https://wiki.qemu.org/Contribute/SubmitAPatch#Participating_in_Code_Review
and:
https://wiki.qemu.org/Contribute/SubmitAPatch#Return_the_favorMy 2 cents...
Regards,
Phil.
Quoted 3 lines
Thanks,
Aleksandar
Signature · 1 line
Eduardo
Re: Bottleneck problem to merge Python patches
In reply to Philippe Mathieu-DaudéQuoted 29 lines · Philippe Mathieu-Daudé
Hi Aleksandar,
(extending the Cc list to broader audience)
Quoted 9 lines · Aleksandar Markovic
On Friday, August 28, 2020, Ahmed Karaman <[email protected] <mailto:[email protected]>> wrote:
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.---
Quoted 8 lines
Hello, folks.
This series seems forgotten. Can some of you perhaps take a look, review, and possibly integrate some patches in a pull request?
Ahmed invested quite of time to improve the functionality and quality of scripts, and they are truly useful for developers interested in performance measurement.
The Python patches are indeed being merged slowly, but are not forgotten :) Eduardo sent just a pull request *yesterday* for patches he had queued *before* QEMU 5.1, that is more than 1 year ago!
https://www.mail-archive.com/[email protected]/msg742228.html
Quoted 6 lines
I hope he will be able to process the other Python patches sent the last 12 months. He raised the problem few month ago saying he was overwhelmed and was looking for help from the community. Cleber helped a pair of times, I helped once, but then nobody popped up volunteering.
Yes, Python patches are not forgotten... they have been haunting me every day and night for the last few months. I have a *lot* of blame to take here, so my sincere apologies.
One word of hope is that my resources (which were exhausted during the last months towards the Avocado 82.0 LTS release, released last week) are now freed. And *a lot* of the Avocado features were eyeing QEMU, so it's just natural that I will (because I want and have to) get back to being more active and responsive.
Quoted 8 lines
I agree this is a community problem, nobody wants to become the de-facto Python maintainer. Current maintainers are already busy solving problem with low-level languages such C.
As a project, QEMU is evolving, using more and more Python, switched to Meson, we might have Rust code too. Learning that for current maintainers takes time. I guess we lack new contributors/maintainers with interest in Python & QEMU.
Quoted 6 lines
This is my simple/rough analysis, as John had the same problem 2/3 months ago, his patches were on the list unreviewed for various weeks. Same problem with Avocado patches, Lukas sent a series a bit before Ahmed and it is still unreviewed:
https://www.mail-archive.com/[email protected]/msg737272.html
ACK. This is a good real example of my previous abstract explanation above.
So, besides me getting back with more resources to review and maintain the Python pathes, I have also requested Willian Rampazzo (thanks!) to keep an eye on those patches and help with reviews.
Quoted 12 lines
Alex Bennée recently said:
"review time is the currency of qemu-devel ;-)"
We might be failing at explaining new contributors the importance of peer-review, as this helps a lot maintainers. It is described in the wiki but maybe new contributors don't read it, we don't remember them to:
https://wiki.qemu.org/Contribute/SubmitAPatch#Participating_in_Code_Review
and:
https://wiki.qemu.org/Contribute/SubmitAPatch#Return_the_favor
Quoted 6 lines
My 2 cents...
Regards,
Phil.
So while more help is always welcome, I do sincerely hope with myself and Willian back into the loop, the Python patches won't lag behind anymore.
And thanks for raising the awareness of the issue Phil!
- Cleber.
Quoted 4 lines
Quoted 3 lines
Thanks,
Aleksandar
Re: [PATCH 1/9] scripts/performance: Refactor topN_perf.py
In reply to Ahmed KaramanI realize this review comes well after you are no longer being paid to work on this, so I am offering my time to help polish your patches if you would like.
Quoted 3 lines · Ahmed Karaman
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.
I had meant to maybe consider using some helper functions so that you didn't need to rename the globals, for instance:
Quoted 46 lines
Signed-off-by Ahmed Karaman <[email protected]>--- scripts/performance/topN_perf.py | 174 +++++++++++++++---------------- 1 file changed, 87 insertions(+), 87 deletions(-)scripts/performance/topN_perf.py +5
-29@@ -1,72 +1,77 @@ #!/usr/bin/env python3 -# Print the top N most executed functions in QEMU using perf. -# Syntax: -# topN_perf.py [-h] [-n] <number of displayed top functions> -- \ -# <qemu executable> [<qemu executable options>] \ -# <target executable> [<target execurable options>] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm -# > -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman <[email protected]> -# Copyright (C) 2020 Aleksandar Markovic <[email protected]> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <https://www.gnu.org/licenses/>. +""" +Print the top N most executed functions in QEMU using perf. + +Syntax: +topN_perf.py [-h] [-n <number of displayed top functions>] -- \
scripts/performance/topN_perf.py line 17
Quoted 10 lines
scripts/performance/topN_perf.py +10
-0+ <qemu executable> [<qemu executable options>] \ + <target executable> [<target execurable options>] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_perf.py -n 20 -- qemu-arm coulomb_double-arm +
Based on discussion we've had upstream since you sent this, I think we will be keeping license and authorship information out of the docstrings, so this part can stay a comment.
Quoted 24 lines
scripts/performance/topN_perf.py +19
-0+This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" import argparse import os import subprocess import sys +import tempfile
scripts/performance/topN_perf.py line 49
Quoted 10 lines
scripts/performance/topN_perf.py +2
-2# Parse the command line arguments -parser = argparse.ArgumentParser( - usage='topN_perf.py [-h] [-n] <number of displayed top functions > -- ' +PARSER = argparse.ArgumentParser( + usage='topN_perf.py [-h] [-n <number of displayed top functions>] -- ' '<qemu executable> [<qemu executable options>] ' '<target executable> [<target executable options>]')
This is a little odd; generally we can avoid having such globals by making a main() function that defines a parser as a local instead.
e.g.,
def main():
parser = ...
parser.add_argument(...)
args = parser.parse_args()
...
return 0if __name__ == '__main__':
sys.exit(main())
Quoted 139 lines
scripts/performance/topN_perf.py +51
-55-parser.add_argument('-n', dest='top', type=int, default=25, +PARSER.add_argument('-n', dest='top', type=int, default=25, help='Specify the number of top functions to print.') -parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) +PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) -args = parser.parse_args() +ARGS = PARSER.parse_args() # Extract the needed variables from the args -command = args.command -top = args.top +COMMAND = ARGS.command +TOP = ARGS.top # Insure that perf is installed -check_perf_presence = subprocess.run(["which", "perf"], - stdout=subprocess.DEVNULL) -if check_perf_presence.returncode: +CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"], + stdout=subprocess.DEVNULL, + check=False) +if CHECK_PERF_PRESENCE.returncode: sys.exit("Please install perf before running the script!") # Insure user has previllage to run perf -check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"], +CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"], stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL) -if check_perf_executability.returncode: - sys.exit( -""" + stderr=subprocess.DEVNULL, + check=False) +if CHECK_PERF_EXECUTABILITY.returncode: + sys.exit(""" Error: You may not have permission to collect stats.@@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conf too, e.g.: kernel.perf_event_paranoid = -1 * Alternatively, you can run this script under sudo privileges. -""" -) - -# Run perf record -perf_record = subprocess.run((["perf", "record", "--output=/tmp/perf.data"] + - command), - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) -if perf_record.returncode: - os.unlink('/tmp/perf.data') - sys.exit(perf_record.stderr.decode("utf-8")) - -# Save perf report output to /tmp/perf_report.out -with open("/tmp/perf_report.out", "w") as output: - perf_report = subprocess.run( - ["perf", "report", "--input=/tmp/perf.data", "--stdio"], - stdout=output, - stderr=subprocess.PIPE) - if perf_report.returncode: - os.unlink('/tmp/perf.data') - output.close() - os.unlink('/tmp/perf_report.out') - sys.exit(perf_report.stderr.decode("utf-8")) - -# Read the reported data to functions[] -functions = [] -with open("/tmp/perf_report.out", "r") as data: - # Only read lines that are not comments (comments start with #) - # Only read lines that are not empty - functions = [line for line in data.readlines() if line and line[0] - != '#' and line[0] != "\n"] - -# Limit the number of top functions to "top" -number_of_top_functions = top if len(functions) > top else len(functions) - -# Store the data of the top functions in top_functions[] -top_functions = functions[:number_of_top_functions] +""") + +# Run perf and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + RECORD_PATH = os.path.join(tmpdir, "record.data") + REPORT_PATH = os.path.join(tmpdir, "report.txt") + + PERF_RECORD = subprocess.run((["perf", "record", "--output="+RECORD_PATH] + + COMMAND), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if PERF_RECORD.returncode: + sys.exit(PERF_RECORD.stderr.decode("utf-8")) + + with open(REPORT_PATH, "w") as output: + PERF_REPORT = subprocess.run( + ["perf", "report", "--input="+RECORD_PATH, "--stdio"], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if PERF_REPORT.returncode: + sys.exit(PERF_REPORT.stderr.decode("utf-8")) + + # Save the reported data to FUNCTIONS[] + with open(REPORT_PATH, "r") as data: + # Only read lines that are not comments (comments start with #) + # Only read lines that are not empty + FUNCTIONS = [line for line in data.readlines() if line and + line[0] != '#' and line[0] != "\n"] + +# Limit the number of top functions to "TOP" +NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS) + +# Store the data of the top functions in TOP_FUNCTIONS[] +TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS] # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',@@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', '-' * 25)) # Print top N functions -for (index, function) in enumerate(top_functions, start=1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=1): function_data = function.split() function_percentage = function_data[0] function_name = function_data[-1]@@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions, start=1): function_percentage, function_name, function_invoker)) - -# Remove intermediate files -os.unlink('/tmp/perf.data') -os.unlink('/tmp/perf_report.out')
Re: [PATCH 1/9] scripts/performance: Refactor topN_perf.py
Quoted 1 line · John Snow
I realize this review comes well after you are no longer being paid to work on this, so I am offering my time to help polish your patches if you would like.
Actually, I see now that you are adding your name to the MAINTAINERS file here, so I suspect you probably rather want to be more involved than not.
I cleaned up patch 1/9 provisionally with my own style preferences, but it's all just style stuff, and it's mostly things I wouldn't actually require you to do (...I went way overboard.)
https://gitlab.com/jsnow/qemu/-/commit/c66a4a6ca8ccc3d406b92796935f92057bf1e48d
What I'd recommend for your cleanup is actually *much* simpler;
Use pylint 2.6.0 and flake8 3.8.3:
Quoted 1 line
pip3 install --user pylint==2.6.0 flake8==3.8.3
flake8's default settings should be pretty good, but pylint has a lot of warnings you can ignore.
In particular, it's OK to use script-style python (Scripts with a #!/usr/bin/env python3, and where you do not use python functions to avoid side-effects that occur on 'import'.) In this case, IGNORE any of pylint's warnings telling you that you have too many lines, that you need to UPPERCASE variable names, etc. It just hurts readability here.
So I'd actually ask that you revise these patches to remove all of the UPPERCASE variable names, and then check your code with these:
flake8 topN_perf.py
pylint --disable=invalid-name topN_perf.py
Use your best judgment -- If something seems like it looks worse, it probably is. If in doubt, please reach out and ask.
--js
PATCH 2/9 scripts/performance: Refactor topN_callgrind.py
In reply to Ahmed Karaman - Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.
--- scripts/performance/topN_callgrind.py | 169 +++++++++++++------------- 1 file changed, 87 insertions(+), 82 deletions(-)
scripts/performance/topN_callgrind.py +87 -82
@@ -1,113 +1,122 @@ #!/usr/bin/env python3 -# Print the top N most executed functions in QEMU using callgrind. -# Syntax: -# topN_callgrind.py [-h] [-n] <number of displayed top functions> -- \ -# <qemu executable> [<qemu executable options>] \ -# <target executable> [<target execurable options>] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman <[email protected]> -# Copyright (C) 2020 Aleksandar Markovic <[email protected]> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <https://www.gnu.org/licenses/>. +""" +Print the top N most executed functions in QEMU using callgrind. + +Syntax: +topN_callgrind.py [-h] [-n <number of displayed top functions>] -- \ + <qemu executable> [<qemu executable options>] \ + <target executable> [<target execurable options>] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" import argparse import os import subprocess import sys +import tempfile # Parse the command line arguments -parser = argparse.ArgumentParser( - usage='topN_callgrind.py [-h] [-n] <number of displayed top functions> -- ' +PARSER = argparse.ArgumentParser( + usage='topN_callgrind.py [-h] [-n] <number of displayed top functions> -- ' '<qemu executable> [<qemu executable options>] ' '<target executable> [<target executable options>]') -parser.add_argument('-n', dest='top', type=int, default=25, +PARSER.add_argument('-n', dest='top', type=int, default=25, help='Specify the number of top functions to print.') -parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) +PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) -args = parser.parse_args() +ARGS = PARSER.parse_args() # Extract the needed variables from the args -command = args.command -top = args.top +COMMAND = ARGS.command +TOP = ARGS.top # Insure that valgrind is installed -check_valgrind_presence = subprocess.run(["which", "valgrind"], - stdout=subprocess.DEVNULL) -if check_valgrind_presence.returncode: +CHECK_VALGRIND_PRESENCE = subprocess.run(["which", "valgrind"], + stdout=subprocess.DEVNULL, + check=False) +if CHECK_VALGRIND_PRESENCE.returncode: sys.exit("Please install valgrind before running the script!") -# Run callgrind -callgrind = subprocess.run(( - ["valgrind", "--tool=callgrind", "--callgrind-out-file=/tmp/callgrind.data"] - + command), - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) -if callgrind.returncode: - sys.exit(callgrind.stderr.decode("utf-8")) - -# Save callgrind_annotate output to /tmp/callgrind_annotate.out -with open("/tmp/callgrind_annotate.out", "w") as output: - callgrind_annotate = subprocess.run(["callgrind_annotate", - "/tmp/callgrind.data"], - stdout=output, - stderr=subprocess.PIPE) - if callgrind_annotate.returncode: - os.unlink('/tmp/callgrind.data') - output.close() - os.unlink('/tmp/callgrind_annotate.out') - sys.exit(callgrind_annotate.stderr.decode("utf-8")) - -# Read the callgrind_annotate output to callgrind_data[] -callgrind_data = [] -with open('/tmp/callgrind_annotate.out', 'r') as data: - callgrind_data = data.readlines() +# Run callgrind and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + CALLGRIND_DATA_PATH = os.path.join(tmpdir, "callgrind.data") + ANNOTATE_OUT_PATH = os.path.join(tmpdir, "callgrind_annotate.out") + + # Run callgrind + CALLGRIND = subprocess.run((["valgrind", + "--tool=callgrind", + "--callgrind-out-file="+CALLGRIND_DATA_PATH] + + COMMAND), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if CALLGRIND.returncode: + sys.exit(CALLGRIND.stderr.decode("utf-8")) + + with open(ANNOTATE_OUT_PATH, "w") as output: + CALLGRIND_ANNOTATE = subprocess.run(["callgrind_annotate", + CALLGRIND_DATA_PATH], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if CALLGRIND_ANNOTATE.returncode: + sys.exit(CALLGRIND_ANNOTATE.stderr.decode("utf-8")) + + # Read the callgrind_annotate output to CALLGRIND_DATA[] + CALLGRIND_DATA = [] + with open(ANNOTATE_OUT_PATH, 'r') as data: + CALLGRIND_DATA = data.readlines() # Line number with the total number of instructions -total_instructions_line_number = 20 +TOTAL_INSTRUCTIONS_LINE_NO = 20 # Get the total number of instructions -total_instructions_line_data = callgrind_data[total_instructions_line_number] -total_number_of_instructions = total_instructions_line_data.split(' ')[0] -total_number_of_instructions = int( - total_number_of_instructions.replace(',', '')) +TOTAL_INSTRUCTIONS_LINE_DATA = CALLGRIND_DATA[TOTAL_INSTRUCTIONS_LINE_NO] +TOTAL_NUMBER_OF_INSTRUCTIONS = TOTAL_INSTRUCTIONS_LINE_DATA.split(' ')[0] +TOTAL_NUMBER_OF_INSTRUCTIONS = int( + TOTAL_NUMBER_OF_INSTRUCTIONS.replace(',', '')) # Line number with the top function -first_func_line = 25 +FIRST_FUNC_LINE = 25 # Number of functions recorded by callgrind, last two lines are always empty -number_of_functions = len(callgrind_data) - first_func_line - 2 +NUMBER_OF_FUNCTIONS = len(CALLGRIND_DATA) - FIRST_FUNC_LINE - 2 # Limit the number of top functions to "top" -number_of_top_functions = (top if number_of_functions > - top else number_of_functions) +NUMBER_OF_TOP_FUNCTIONS = (TOP if NUMBER_OF_FUNCTIONS > + TOP else NUMBER_OF_FUNCTIONS) # Store the data of the top functions in top_functions[] -top_functions = callgrind_data[first_func_line: - first_func_line + number_of_top_functions] +TOP_FUNCTIONS = CALLGRIND_DATA[FIRST_FUNC_LINE: + FIRST_FUNC_LINE + NUMBER_OF_TOP_FUNCTIONS] # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
@@ -121,12 +130,12 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', )) # Print top N functions -for (index, function) in enumerate(top_functions, start=1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=1): function_data = function.split() # Calculate function percentage function_instructions = float(function_data[0].replace(',', '')) function_percentage = (function_instructions / - total_number_of_instructions)*100 + TOTAL_NUMBER_OF_INSTRUCTIONS)*100 # Get function name and source files path function_source_file, function_name = function_data[1].split(':') # Print extracted data
@@ -134,7 +143,3 @@ for (index, function) in enumerate(top_functions, start=1): round(function_percentage, 3), function_name, function_source_file)) - -# Remove intermediate files -os.unlink('/tmp/callgrind.data') -os.unlink('/tmp/callgrind_annotate.out')
Signature · 1 line
2.17.1
PATCH 2/9 scripts/performance: Refactor topN_callgrind.py
Quoted 4 lines · Ahmed Karaman
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.
Reply
Reply all
Forward
View Gmail in: *Mobile* | Older version
<https://mail.google.com/mail/mu/mp/966/#> | Desktop
<https://mail.google.com/mail/mu/mp/966/#>
© 2020 Google
Quoted 248 lines
Signed-off-by Ahmed Karaman <[email protected]>--- scripts/performance/topN_callgrind.py | 169 +++++++++++++------------- 1 file changed, 87 insertions(+), 82 deletions(-)scripts/performance/topN_callgrind.py +0
-17@@ -1,113 +1,122 @@ #!/usr/bin/env python3 -# Print the top N most executed functions in QEMU using callgrind. -# Syntax: -# topN_callgrind.py [-h] [-n] <number of displayed top functions> -- \ -# <qemu executable> [<qemu executable options>] \ -# <target executable> [<target execurable options>] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman <[email protected]> -# Copyright (C) 2020 Aleksandar Markovic <[email protected]om>
scripts/performance/topN_callgrind.py +34
-15-# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <https://www.gnu.org/licenses/>. +""" +Print the top N most executed functions in QEMU using callgrind. + +Syntax: +topN_callgrind.py [-h] [-n <number of displayed top functions>] -- \ + <qemu executable> [<qemu executable options>] \ + <target executable> [<target execurable options>] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" import argparse import os import subprocess import sys +import tempfile # Parse the command line arguments -parser = argparse.ArgumentParser( - usage='topN_callgrind.py [-h] [-n] <number of displayed topfunctions> -- '
scripts/performance/topN_callgrind.py +2
-0+PARSER = argparse.ArgumentParser( + usage='topN_callgrind.py [-h] [-n] <number of displayed topfunctions> -- '
'<qemu executable> [<qemu executable options>] ' '<target executable> [<target executable options>]')scripts/performance/topN_callgrind.py +1
-2-parser.add_argument('-n', dest='top', type=int, default=25, +PARSER.add_argument('-n', dest='top', type=int, default=25, help='Specify the number of top functions to print.') -parser.add_argument('command', type=str, nargs='+',help=argparse.SUPPRESS) +PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
scripts/performance/topN_callgrind.py +7
-9-args = parser.parse_args() +ARGS = PARSER.parse_args() # Extract the needed variables from the args -command = args.command -top = args.top +COMMAND = ARGS.command +TOP = ARGS.top # Insure that valgrind is installed -check_valgrind_presence = subprocess.run(["which", "valgrind"], - stdout=subprocess.DEVNULL) -if check_valgrind_presence.returncode: +CHECK_VALGRIND_PRESENCE = subprocess.run(["which", "valgrind"], + stdout=subprocess.DEVNULL, + check=False) +if CHECK_VALGRIND_PRESENCE.returncode: sys.exit("Please install valgrind before running the script!") -# Run callgrind -callgrind = subprocess.run(( - ["valgrind", "--tool=callgrind", "--callgrind-out-file=/tmp/callgrind.data"]
scripts/performance/topN_callgrind.py +9
-22- + command), - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) -if callgrind.returncode: - sys.exit(callgrind.stderr.decode("utf-8")) - -# Save callgrind_annotate output to /tmp/callgrind_annotate.out -with open("/tmp/callgrind_annotate.out", "w") as output: - callgrind_annotate = subprocess.run(["callgrind_annotate", - "/tmp/callgrind.data"], - stdout=output, - stderr=subprocess.PIPE) - if callgrind_annotate.returncode: - os.unlink('/tmp/callgrind.data') - output.close() - os.unlink('/tmp/callgrind_annotate.out') - sys.exit(callgrind_annotate.stderr.decode("utf-8")) - -# Read the callgrind_annotate output to callgrind_data[] -callgrind_data = [] -with open('/tmp/callgrind_annotate.out', 'r') as data: - callgrind_data = data.readlines() +# Run callgrind and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + CALLGRIND_DATA_PATH = os.path.join(tmpdir, "callgrind.data") + ANNOTATE_OUT_PATH = os.path.join(tmpdir, "callgrind_annotate.out") + + # Run callgrind + CALLGRIND = subprocess.run((["valgrind", + "--tool=callgrind", + "--callgrind-out-file="+CALLG
RIND_DATA_PATH]scripts/performance/topN_callgrind.py +21
-2+ + COMMAND), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if CALLGRIND.returncode: + sys.exit(CALLGRIND.stderr.decode("utf-8")) + + with open(ANNOTATE_OUT_PATH, "w") as output: + CALLGRIND_ANNOTATE = subprocess.run(["callgrind_annotate", + CALLGRIND_DATA_PATH], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if CALLGRIND_ANNOTATE.returncode: + sys.exit(CALLGRIND_ANNOTATE.stderr.decode("utf-8")) + + # Read the callgrind_annotate output to CALLGRIND_DATA[] + CALLGRIND_DATA = [] + with open(ANNOTATE_OUT_PATH, 'r') as data: + CALLGRIND_DATA = data.readlines() # Line number with the total number of instructions -total_instructions_line_number = 20 +TOTAL_INSTRUCTIONS_LINE_NO = 20 # Get the total number of instructions -total_instructions_line_data = callgrind_data[total_instructi
ons_line_number]scripts/performance/topN_callgrind.py +5
-4-total_number_of_instructions = total_instructions_line_data.split(' ')[0] -total_number_of_instructions = int( - total_number_of_instructions.replace(',', '')) +TOTAL_INSTRUCTIONS_LINE_DATA = CALLGRIND_DATA[TOTAL_INSTRUCTIONS_LINE_NO] +TOTAL_NUMBER_OF_INSTRUCTIONS = TOTAL_INSTRUCTIONS_LINE_DATA.split(' ')[0] +TOTAL_NUMBER_OF_INSTRUCTIONS = int( + TOTAL_NUMBER_OF_INSTRUCTIONS.replace(',', '')) # Line number with the top function -first_func_line = 25 +FIRST_FUNC_LINE = 25 # Number of functions recorded by callgrind, last two lines are alwaysempty
scripts/performance/topN_callgrind.py +7
-7-number_of_functions = len(callgrind_data) - first_func_line - 2 +NUMBER_OF_FUNCTIONS = len(CALLGRIND_DATA) - FIRST_FUNC_LINE - 2 # Limit the number of top functions to "top" -number_of_top_functions = (top if number_of_functions > - top else number_of_functions) +NUMBER_OF_TOP_FUNCTIONS = (TOP if NUMBER_OF_FUNCTIONS > + TOP else NUMBER_OF_FUNCTIONS) # Store the data of the top functions in top_functions[] -top_functions = callgrind_data[first_func_line: - first_func_line + number_of_top_functions] +TOP_FUNCTIONS = CALLGRIND_DATA[FIRST_FUNC_LINE: + FIRST_FUNC_LINE + NUMBER_OF_TOP_FUNCTIONS] # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',@@ -121,12 +130,12 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', )) # Print top N functions -for (index, function) in enumerate(top_functions, start=1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=1): function_data = function.split() # Calculate function percentage function_instructions = float(function_data[0].replace(',', '')) function_percentage = (function_instructions / - total_number_of_instructions)*100 + TOTAL_NUMBER_OF_INSTRUCTIONS)*100 # Get function name and source files path function_source_file, function_name = function_data[1].split(':')# Print extracted data
scripts/performance/topN_callgrind.py +0
-0@@ -134,7 +143,3 @@ for (index, function) in enumerate(top_functions,start=1):
round(function_percentage, 3),
function_name,function_source_file))scripts/performance/topN_callgrind.py +0
-5- -# Remove intermediate files -os.unlink('/tmp/callgrind.data') -os.unlink('/tmp/callgrind_annotate.out') --2.17.1
PATCH 3/9 scripts/performance: Refactor dissect.py
In reply to Ahmed Karaman- Apply pylint and flake8 formatting rules to the script.
- Move syntax and usage exmaple to main() docstring.
- Update get_jit_line() to only detect the main jit call.
- Use mypy for specifying parameters and return types in functions.
--- scripts/performance/dissect.py | 123 ++++++++++++++++++--------------- 1 file changed, 68 insertions(+), 55 deletions(-)
scripts/performance/dissect.py +68 -55
@@ -1,34 +1,27 @@ #!/usr/bin/env python3 -# Print the percentage of instructions spent in each phase of QEMU -# execution. -# -# Syntax: -# dissect.py [-h] -- <qemu executable> [<qemu executable options>] \ -# <target executable> [<target executable options>] -# -# [-h] - Print the script arguments help message. -# -# Example of usage: -# dissect.py -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman <[email protected]> -# Copyright (C) 2020 Aleksandar Markovic <[email protected]> -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <https://www.gnu.org/licenses/>. +""" +Print the percentage of instructions spent in each phase of QEMU +execution. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" import argparse import os
@@ -36,23 +29,26 @@ import subprocess import sys import tempfile +from typing import List + -def get_JIT_line(callgrind_data): +def get_jit_line(callgrind_data: List[str]) -> int: """ Search for the first instance of the JIT call in the callgrind_annotate output when ran using --tree=caller This is equivalent to the self number of instructions of JIT. Parameters: - callgrind_data (list): callgrind_annotate output + callgrind_data (List[str]): callgrind_annotate output Returns: (int): Line number """ line = -1 - for i in range(len(callgrind_data)): - if callgrind_data[i].strip('\n') and \ - callgrind_data[i].split()[-1] == "[???]": + for (i, callgrind_datum) in enumerate(callgrind_data): + if callgrind_datum.strip('\n') and \ + callgrind_datum.split()[-1] == "[???]" and \ + callgrind_datum.split()[1] == "*": line = i break if line == -1:
@@ -61,6 +57,18 @@ def get_JIT_line(callgrind_data): def main(): + """ + Parse the command line arguments then start the execution. + Syntax: + dissect.py [-h] -- <qemu executable> [<qemu executable options>] \ + <target executable> [<target executable options>] + + [-h] - Print the script arguments help message. + + Example of usage: + dissect.py -- qemu-arm coulomb_double-arm + """ + # Parse the command line arguments parser = argparse.ArgumentParser( usage='dissect.py [-h] -- '
@@ -76,7 +84,7 @@ def main(): # Insure that valgrind is installed check_valgrind = subprocess.run( - ["which", "valgrind"], stdout=subprocess.DEVNULL) + ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False) if check_valgrind.returncode: sys.exit("Please install valgrind before running the script.")
@@ -93,7 +101,8 @@ def main(): "--callgrind-out-file=" + data_path] + command), stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) + stderr=subprocess.PIPE, + check=False) if callgrind.returncode: sys.exit(callgrind.stderr.decode("utf-8"))
@@ -102,7 +111,8 @@ def main(): callgrind_annotate = subprocess.run( ["callgrind_annotate", data_path, "--tree=caller"], stdout=output, - stderr=subprocess.PIPE) + stderr=subprocess.PIPE, + check=False) if callgrind_annotate.returncode: sys.exit(callgrind_annotate.stderr.decode("utf-8"))
@@ -120,25 +130,28 @@ def main(): total_instructions = int(total_instructions.replace(',', '')) # Line number with the JIT self number of instructions - JIT_self_instructions_line_number = get_JIT_line(callgrind_data) + jit_self_instructions_line_number = get_jit_line(callgrind_data) # Get the JIT self number of instructions - JIT_self_instructions_line_data = \ - callgrind_data[JIT_self_instructions_line_number] - JIT_self_instructions = JIT_self_instructions_line_data.split()[0] - JIT_self_instructions = int(JIT_self_instructions.replace(',', '')) + jit_self_instructions_line_data = \ + callgrind_data[jit_self_instructions_line_number] + jit_self_instructions = jit_self_instructions_line_data.split()[0] + jit_self_instructions = int(jit_self_instructions.replace(',', '')) # Line number with the JIT self + inclusive number of instructions - # It's the line above the first JIT call when running with --tree=caller - JIT_total_instructions_line_number = JIT_self_instructions_line_number-1 + # It's the line above the first JIT call when running with + # --tree=caller + jit_total_instructions_line_number = \ + jit_self_instructions_line_number-1 # Get the JIT self + inclusive number of instructions - JIT_total_instructions_line_data = \ - callgrind_data[JIT_total_instructions_line_number] - JIT_total_instructions = JIT_total_instructions_line_data.split()[0] - JIT_total_instructions = int(JIT_total_instructions.replace(',', '')) + jit_total_instructions_line_data = \ + callgrind_data[jit_total_instructions_line_number] + jit_total_instructions = jit_total_instructions_line_data.split()[0] + jit_total_instructions = int(jit_total_instructions.replace(',', '')) # Calculate number of instructions in helpers and code generation - helpers_instructions = JIT_total_instructions-JIT_self_instructions - code_generation_instructions = total_instructions-JIT_total_instructions + helpers_instructions = jit_total_instructions-jit_self_instructions + code_generation_instructions = \ + total_instructions-jit_total_instructions # Print results (Insert commas in large numbers) # Print total number of instructions
@@ -149,12 +162,12 @@ def main(): print('{:<20}{:>20}\t{:>6.3f}%'. format("Code Generation:", format(code_generation_instructions, ","), - (code_generation_instructions / total_instructions) * 100)) - # Print JIT instructions and percentage + (code_generation_instructions/total_instructions)*100)) + # Print jit instructions and percentage print('{:<20}{:>20}\t{:>6.3f}%'. - format("JIT Execution:", - format(JIT_self_instructions, ","), - (JIT_self_instructions / total_instructions) * 100)) + format("jit Execution:", + format(jit_self_instructions, ","), + (jit_self_instructions / total_instructions) * 100)) # Print helpers instructions and percentage print('{:<20}{:>20}\t{:>6.3f}%'. format("Helpers:",
Signature · 1 line
2.17.1
Re: [PATCH 3/9] scripts/performance: Refactor dissect.py
Quoted 6 lines · Ahmed Karaman
- Apply pylint and flake8 formatting rules to the script.
- Move syntax and usage exmaple tomain()docstring.
- Updateget_jit_line()to only detect the main jit call.
- Use mypy for specifying parameters and return types in functions.
Quoted 227 lines
Signed-off-by Ahmed Karaman <[email protected]>--- scripts/performance/dissect.py | 123 ++++++++++++++++++--------------- 1 file changed, 68 insertions(+), 55 deletions(-)scripts/performance/dissect. renamed from scripts/performance/dissect.py +0
-0py
indexbf24f50922..d4df884b75100755scripts/performance/dissect.py +0
-16@@ -1,34 +1,27 @@ #!/usr/bin/env python3 -# Print the percentage of instructions spent in each phase of QEMU -# execution. -# -# Syntax: -# dissect.py [-h] -- <qemu executable> [<qemu executable options>] \ -# <target executable> [<target executable options>] -# -# [-h] - Print the script arguments help message. -# -# Example of usage: -# dissect.py -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman <[email protected]> -# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
scripts/performance/dissect.py +48
-25-# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see <https://www.gnu.org/licenses/>. +""" +Print the percentage of instructions spent in each phase of QEMU +execution. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" import argparse import os@@ -36,23 +29,26 @@ import subprocess import sys import tempfile +from typing import List + -def get_JIT_line(callgrind_data): +def get_jit_line(callgrind_data: List[str]) -> int: """ Search for the first instance of the JIT call in the callgrind_annotate output when ran using --tree=caller This is equivalent to the self number of instructions of JIT. Parameters: - callgrind_data (list): callgrind_annotate output + callgrind_data (List[str]): callgrind_annotate output Returns: (int): Line number """ line = -1 - for i in range(len(callgrind_data)): - if callgrind_data[i].strip('\n') and \ - callgrind_data[i].split()[-1] == "[???]": + for (i, callgrind_datum) in enumerate(callgrind_data): + if callgrind_datum.strip('\n') and \ + callgrind_datum.split()[-1] == "[???]" and \ + callgrind_datum.split()[1] == "*": line = i break if line == -1:@@ -61,6 +57,18 @@ def get_JIT_line(callgrind_data): def main(): + """ + Parse the command line arguments then start the execution. + Syntax: + dissect.py [-h] -- <qemu executable> [<qemu executable options>] \ + <target executable> [<target executable options>] + + [-h] - Print the script arguments help message. + + Example of usage: + dissect.py -- qemu-arm coulomb_double-arm + """ + # Parse the command line arguments parser = argparse.ArgumentParser( usage='dissect.py [-h] -- '@@ -76,7 +84,7 @@ def main(): # Insure that valgrind is installed check_valgrind = subprocess.run( - ["which", "valgrind"], stdout=subprocess.DEVNULL) + ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False) if check_valgrind.returncode: sys.exit("Please install valgrind before running the script.")@@ -93,7 +101,8 @@ def main(): "--callgrind-out-file=" + data_path] + command), stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) + stderr=subprocess.PIPE, + check=False) if callgrind.returncode: sys.exit(callgrind.stderr.decode("utf-8"))@@ -102,7 +111,8 @@ def main(): callgrind_annotate = subprocess.run( ["callgrind_annotate", data_path, "--tree=caller"], stdout=output, - stderr=subprocess.PIPE) + stderr=subprocess.PIPE, + check=False) if callgrind_annotate.returncode: sys.exit(callgrind_annotate.stderr.decode("utf-8"))@@ -120,25 +130,28 @@ def main(): total_instructions = int(total_instructions.replace(',', '')) # Line number with the JIT self number of instructions - JIT_self_instructions_line_number = get_JIT_line(callgrind_data) + jit_self_instructions_line_number = get_jit_line(callgrind_data) # Get the JIT self number of instructions - JIT_self_instructions_line_data = \ - callgrind_data[JIT_self_instructions_line_number] - JIT_self_instructions = JIT_self_instructions_line_data.split()[0]
-JIT_self_instructions= int(JIT_self_instructions.replace(',',
''))scripts/performance/dissect.py +3
-0+ jit_self_instructions_line_data = \ + callgrind_data[jit_self_instructions_line_number] + jit_self_instructions = jit_self_instructions_line_data.split()[0]
+jit_self_instructions= int(jit_self_instructions.replace(',',
''))# Line number with the JIT self + inclusive number of instructions
scripts/performance/dissect.py +0
-3- # It's the line above the first JIT call when running with --tree=caller - JIT_total_instructions_line_number = JIT_self_instructions_line_number-1
scripts/performance/dissect.py +4
-3+ # It's the line above the first JIT call when running with + # --tree=caller + jit_total_instructions_line_number = \ + jit_self_instructions_line_number-1 # Get the JIT self + inclusive number of instructions - JIT_total_instructions_line_data = \ - callgrind_data[JIT_total_instructions_line_number] - JIT_total_instructions = JIT_total_instructions_line_data.split()[0]
-JIT_total_instructions= int(JIT_total_instructions.replace(',',
''))scripts/performance/dissect.py +3
-0+ jit_total_instructions_line_data = \ + callgrind_data[jit_total_instructions_line_number] + jit_total_instructions = jit_total_instructions_line_data.split()[0]
+jit_total_instructions= int(jit_total_instructions.replace(',',
''))# Calculate number of instructions in helpers and code generation
-helpers_instructions= JIT_total_instructions-JIT_self_instructions
-code_generation_instructions= total_instructions-JIT_total_
instructions
+helpers_instructions= jit_total_instructions-jit_self_instructionsscripts/performance/dissect.py +7
-5+ code_generation_instructions = \ + total_instructions-jit_total_instructions # Print results (Insert commas in large numbers) # Print total number of instructions@@ -149,12 +162,12 @@ def main(): print('{:<20}{:>20}\t{:>6.3f}%'. format("Code Generation:", format(code_generation_instructions, ","), - (code_generation_instructions / total_instructions) * 100)) - # Print JIT instructions and percentage + (code_generation_instructions/ total_instructions)*100)) + # Print jit instructions and percentage print('{:<20}{:>20}\t{:>6.3f}%'. - format("JIT Execution:", - format(JIT_self_instructions, ","), - (JIT_self_instructions / total_instructions) * 100)) + format("jit Execution:", + format(jit_self_instructions, ","), + (jit_self_instructions / total_instructions) * 100)) # Print helpers instructions and percentageprint('{:<20}{:>20}\t{:>6.3f}%'. format("Helpers:",--
2.17.1
PATCH 4/9 scripts/performance: Add list_fn_callees.py script
In reply to Ahmed KaramanPython script that prints the callees of a given list of QEMU functions.
Syntax:list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \
<qemu executable> [<qemu executable options>] \
<target executable> [<target executable options>]
[-h] - Print the script arguments help message. -f FUNCTION [FUNCTION ...] - List of function names
Example of usage:list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \
qemu-mips coulomb_double-mips -n10
Example output:
Total number of instructions: 108,952,851
Callees of helper_float_sub_d:
No. Instructions Percentage Calls Ins/Call Function Name Source File --- ------------ ---------- ------ -------- ------------- ---------------
1 153,160 0.141% 1,305 117 float64_sub <qemu>/fpu/softfloat.c
Callees of helper_float_mul_d:
No. Instructions Percentage Calls Ins/Call Function Name Source File --- ------------ ---------- ------ -------- ------------- ---------------
1 131,137 0.120% 1,014 129 float64_mul <qemu>/fpu/softfloat.c
--- scripts/performance/list_fn_callees.py | 245 +++++++++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100755 scripts/performance/list_fn_callees.py
scripts/performance/list_fn_callees.py new file +245 -0
@@ -0,0 +1,245 @@ +#!/usr/bin/env python3 + +""" +Print the callees of a given list of QEMU functions. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" + +import argparse +import os +import subprocess +import sys +import tempfile + +from typing import List, Union + + +def find_function_lines(function_name: str, + callgrind_data: List[str]) -> List[int]: + """ + Search for the line with the function name in the + callgrind_annotate output when ran using --tre=calling. + All the function callees should be listed after that line. + + Parameters: + function_name (string): The desired function name to print its callees + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (List[int]): List of function line numbers + """ + lines = [] + for (i, callgrind_datum) in enumerate(callgrind_data): + split_line = callgrind_datum.split() + if len(split_line) > 2 and \ + split_line[1] == "*" and \ + split_line[2].split(":")[-1] == function_name: + # Function might be in the callgrind_annotate output more than + # once, so don't break after finding an instance + if callgrind_data[i + 1] != "\n": + # Only append the line number if the found instance has + # callees + lines.append(i) + return lines + + +def get_function_calles( + function_lines: List[int], + callgrind_data: List[str]) -> List[List[Union[str, int]]]: + """ + Get all callees data for a function given its list of line numbers in + callgrind_annotate output. + + Parameters: + function_lines (List[int]): Line numbers of the function to get its callees + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (List[List[Union[str, int]]]):[[number_of_instructions(int), + callee_name(str), + number_of_calls(int), + source_file(str)], + ...] + """ + callees: List[List[Union[str, int]]] = [] + for function_line in function_lines: + next_callee = function_line + 1 + while callgrind_data[next_callee] != "\n": + split_line = callgrind_data[next_callee].split() + number_of_instructions = int(split_line[0].replace(",", "")) + source_file = split_line[2].split(":")[0] + callee_name = split_line[2].split(":")[1] + number_of_calls = int(split_line[3][1:-2]) + callees.append([number_of_instructions, callee_name, + number_of_calls, source_file]) + next_callee += 1 + return sorted(callees, reverse=True) + + +def main(): + """ + Parse the command line arguments then start execution. + + Syntax: + list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \ + <qemu executable> [<qemu executable options>] \ + <target executable> [<target executable options>] + + [-h] - Print the script arguments help message. + -f FUNCTION [FUNCTION ...] - List of function names + + Example of usage: + list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \ + qemu-mips coulomb_double-mips + """ + + # Parse the command line arguments + parser = argparse.ArgumentParser( + usage="list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- " + "<qemu executable> [<qemu executable options>] " + "<target executable> [<target executable options>]") + + parser.add_argument("-f", dest="function", type=str, + nargs="+", required=True, + help="list of function names to print their callees") + + parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS) + + args = parser.parse_args() + + # Extract the needed variables from the args + command = args.command + function_names = args.function + + # Insure that valgrind is installed + check_valgrind = subprocess.run( + ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False) + if check_valgrind.returncode: + sys.exit("Please install valgrind before running the script.") + + # Save all intermediate files in a temporary directory + with tempfile.TemporaryDirectory() as tmpdirname: + # callgrind output file path + data_path = os.path.join(tmpdirname, "callgrind.data") + # callgrind_annotate output file path + annotate_out_path = os.path.join(tmpdirname, "callgrind_annotate.out") + + # Run callgrind + callgrind = subprocess.run((["valgrind", + "--tool=callgrind", + "--callgrind-out-file=" + data_path] + + command), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if callgrind.returncode: + sys.exit(callgrind.stderr.decode("utf-8")) + + # Save callgrind_annotate output + with open(annotate_out_path, "w") as output: + callgrind_annotate = subprocess.run(["callgrind_annotate", + data_path, + "--threshold=100", + "--tree=calling"], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if callgrind_annotate.returncode: + sys.exit(callgrind_annotate.stderr.decode("utf-8")) + + # Read the callgrind_annotate output to callgrind_data[] + callgrind_data = [] + with open(annotate_out_path, "r") as data: + callgrind_data = data.readlines() + + # Line number with the total number of instructions + total_instructions_line_number = 20 + # Get the total number of instructions + total_instructions_line_data = \ + callgrind_data[total_instructions_line_number] + total_instructions = total_instructions_line_data.split()[0] + + print("Total number of instructions: {}\n".format(total_instructions)) + + # Remove commas and convert to int + total_instructions = int(total_instructions.replace(",", "")) + + for function_name in function_names: + # Line numbers with the desired function + function_lines = find_function_lines(function_name, callgrind_data) + + if len(function_lines) == 0: + print("Couldn't locate function: {}.\n".format( + function_name)) + continue + + # Get function callees + function_callees = get_function_calles( + function_lines, callgrind_data) + + print("Callees of {}:\n".format(function_name)) + + # Print table header + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "No.", + "Instructions", + "Percentage", + "Calls", + "Ins/Call", + "Function Name", + "Source File") + ) + + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "-" * 4, + "-" * 15, + "-" * 10, + "-" * 15, + "-" * 10, + "-" * 25, + "-" * 30) + ) + + for (index, callee) in enumerate(function_callees, start=1): + instructions = callee[0] + percentage = (callee[0] / total_instructions) * 100 + calls = callee[2] + instruction_per_call = int(callee[0] / callee[2]) + function_name = callee[1] + source_file = callee[3] + # Print extracted data + print("{:>4} {:>15} {:>9.3f}% {:>15} {:>10} {:<25} {}". + format( + index, + format(instructions, ","), + round(percentage, 3), + format(calls, ","), + format(instruction_per_call, ","), + function_name, + source_file) + ) + + print("\n") + + +if __name__ == "__main__": + main()
Signature · 1 line
2.17.1
PATCH 5/9 scripts/performance: Add list_helpers.py script
In reply to Ahmed KaramanPython script that prints executed helpers of a QEMU invocation.
Syntax:list_helpers.py [-h] -- \
<qemu executable> [<qemu executable options>] \
<target executable> [<target executable options>]
[-h] - Print the script arguments help message.
Example of usage:list_helpers.py -- qemu-mips coulomb_double-mips -n10
Example output:
Total number of instructions: 108,933,695
Executed QEMU Helpers:
No. Ins Percent Calls Ins/Call Helper Name Source File --- ------- ------- ------ -------- -------------------- --------------- 1 183,021 0.168% 1,305 140 helper_float_sub_d <qemu>/target/mips/fpu_helper.c 2 177,111 0.163% 770 230 helper_float_madd_d <qemu>/target/mips/fpu_helper.c 3 171,537 0.157% 1,014 169 helper_float_mul_d <qemu>/target/mips/fpu_helper.c 4 157,298 0.144% 2,443 64 helper_lookup_tb_ptr <qemu>/accel/tcg/tcg-runtime.c 5 138,123 0.127% 897 153 helper_float_add_d <qemu>/target/mips/fpu_helper.c 6 47,083 0.043% 207 227 helper_float_msub_d <qemu>/target/mips/fpu_helper.c 7 24,062 0.022% 487 49 helper_cmp_d_lt <qemu>/target/mips/fpu_helper.c 8 22,910 0.021% 150 152 helper_float_div_d <qemu>/target/mips/fpu_helper.c 9 15,497 0.014% 321 48 helper_cmp_d_eq <qemu>/target/mips/fpu_helper.c 10 9,100 0.008% 52 175 helper_float_trunc_w_d <qemu>/target/mips/fpu_helper.c 11 7,059 0.006% 10 705 helper_float_sqrt_d <qemu>/target/mips/fpu_helper.c 12 3,000 0.003% 40 75 helper_cmp_d_ule <qemu>/target/mips/fpu_helper.c 13 2,720 0.002% 20 136 helper_float_cvtd_w <qemu>/target/mips/fpu_helper.c 14 2,477 0.002% 27 91 helper_swl <qemu>/target/mips/op_helper.c 15 2,000 0.002% 40 50 helper_cmp_d_le <qemu>/target/mips/fpu_helper.c 16 1,800 0.002% 40 45 helper_cmp_d_un <qemu>/target/mips/fpu_helper.c 17 1,164 0.001% 12 97 helper_raise_exception_ <qemu>/target/mips/op_helper.c 18 720 0.001% 10 72 helper_cmp_d_ult <qemu>/target/mips/fpu_helper.c 19 560 0.001% 140 4 helper_cfc1 <qemu>/target/mips/fpu_helper.c
--- scripts/performance/list_helpers.py | 221 ++++++++++++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100755 scripts/performance/list_helpers.py
scripts/performance/list_helpers.py new file +221 -0
@@ -0,0 +1,221 @@ +#!/usr/bin/env python3 + +""" +Print the executed helpers of a QEMU invocation. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" + +import argparse +import os +import subprocess +import sys +import tempfile + +from typing import List, Union + + +def find_jit_line(callgrind_data: List[str]) -> int: + """ + Search for the line with the JIT call in the callgrind_annotate + output when ran using --tre=calling. + All the helpers should be listed after that line. + + Parameters: + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (int): Line number of JIT call + """ + line = -1 + for (i, callgrind_datum) in enumerate(callgrind_data): + split_line = callgrind_datum.split() + if len(split_line) > 2 and \ + split_line[1] == "*" and \ + split_line[-1] == "[???]": + line = i + break + return line + + +def get_helpers(jit_line: int, + callgrind_data: List[str]) -> List[List[Union[str, int]]]: + """ + Get all helpers data given the line number of the JIT call. + + Parameters: + jit_line (int): Line number of the JIT call + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (List[List[Union[str, int]]]):[[number_of_instructions(int), + helper_name(str), + number_of_calls(int), + source_file(str)], + ...] + """ + helpers: List[List[Union[str, int]]] = [] + next_helper = jit_line + 1 + while callgrind_data[next_helper] != "\n": + split_line = callgrind_data[next_helper].split() + number_of_instructions = int(split_line[0].replace(",", "")) + source_file = split_line[2].split(":")[0] + callee_name = split_line[2].split(":")[1] + number_of_calls = int(split_line[3][1:-2]) + helpers.append([number_of_instructions, callee_name, + number_of_calls, source_file]) + next_helper += 1 + return sorted(helpers, reverse=True) + + +def main(): + """ + Parse the command line arguments then start execution + + Syntax: + list_helpers.py [-h] -- \ + <qemu executable> [<qemu executable options>] \ + <target executable> [<target executable options>] + + [-h] - Print the script arguments help message. + + Example of usage: + list_helpers.py -- qemu-mips coulomb_double-mips + """ + # Parse the command line arguments + parser = argparse.ArgumentParser( + usage="list_helpers.py [-h] -- " + "<qemu executable> [<qemu executable options>] " + "<target executable> [<target executable options>]") + + parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS) + + args = parser.parse_args() + + # Extract the needed variables from the args + command = args.command + + # Insure that valgrind is installed + check_valgrind = subprocess.run( + ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False) + if check_valgrind.returncode: + sys.exit("Please install valgrind before running the script.") + + # Save all intermediate files in a temporary directory + with tempfile.TemporaryDirectory() as tmpdirname: + # callgrind output file path + data_path = os.path.join(tmpdirname, "callgrind.data") + # callgrind_annotate output file path + annotate_out_path = os.path.join(tmpdirname, "callgrind_annotate.out") + + # Run callgrind + callgrind = subprocess.run((["valgrind", + "--tool=callgrind", + "--callgrind-out-file=" + data_path] + + command), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if callgrind.returncode: + sys.exit(callgrind.stderr.decode("utf-8")) + + # Save callgrind_annotate output + with open(annotate_out_path, "w") as output: + callgrind_annotate = subprocess.run(["callgrind_annotate", + data_path, + "--threshold=100", + "--tree=calling"], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if callgrind_annotate.returncode: + sys.exit(callgrind_annotate.stderr.decode("utf-8")) + + # Read the callgrind_annotate output to callgrind_data[] + callgrind_data = [] + with open(annotate_out_path, "r") as data: + callgrind_data = data.readlines() + + # Line number with the total number of instructions + total_instructions_line_number = 20 + # Get the total number of instructions + total_instructions_line_data = \ + callgrind_data[total_instructions_line_number] + total_instructions = total_instructions_line_data.split()[0] + + print("Total number of instructions: {}\n".format(total_instructions)) + + # Remove commas and convert to int + total_instructions = int(total_instructions.replace(",", "")) + + # Line number with the JIT call + jit_line = find_jit_line(callgrind_data) + if jit_line == -1: + sys.exit("Couldn't locate the JIT call ... Exiting") + + # Get helpers + helpers = get_helpers(jit_line, callgrind_data) + + print("Executed QEMU Helpers:\n") + + # Print table header + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "No.", + "Instructions", + "Percentage", + "Calls", + "Ins/Call", + "Helper Name", + "Source File") + ) + + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "-" * 4, + "-" * 15, + "-" * 10, + "-" * 15, + "-" * 10, + "-" * 25, + "-" * 30) + ) + + for (index, callee) in enumerate(helpers, start=1): + instructions = callee[0] + percentage = (callee[0] / total_instructions) * 100 + calls = callee[2] + instruction_per_call = int(callee[0] / callee[2]) + helper_name = callee[1] + source_file = callee[3] + # Print extracted data + print("{:>4} {:>15} {:>9.3f}% {:>15} {:>10} {:<25} {}". + format( + index, + format(instructions, ","), + round(percentage, 3), + format(calls, ","), + format(instruction_per_call, ","), + helper_name, + source_file) + ) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.17.1
PATCH 6/9 scripts/performance: Add bisect.py script
In reply to Ahmed KaramanPython script that locates the commit that caused a performance degradation or improvement in QEMU using the git bisect command (binary search).
Syntax:bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \
--target TARGET --tool {perf,callgrind} -- \
<target executable> [<target executable options>]
[-h] - Print the script arguments help message -s,--start START - First commit hash in the search range [-e,--end END] - Last commit hash in the search range
(default: Latest commit)
[-q,--qemu QEMU] - QEMU path.
(default: Path to a GitHub QEMU clone)
--target TARGET - QEMU target name
--tool {perf,callgrind} - Underlying tool used for measurements
Example of usage:bisect.py --start=fdd76fecdd --qemu=/path/to/qemu --target=ppc \
--tool=perf -- coulomb_double-ppc -n 1000
Example output:
Start Commit Instructions: 12,710,790,060 End Commit Instructions: 13,031,083,512 Performance Change: -2.458%
Estimated Number of Steps: 10
*****************BISECT STEP 1*****************
Instructions: 13,031,097,790 Status: slow commit *****************BISECT STEP 2***************** Instructions: 12,710,805,265 Status: fast commit *****************BISECT STEP 3***************** Instructions: 13,031,028,053 Status: slow commit *****************BISECT STEP 4***************** Instructions: 12,711,763,211 Status: fast commit *****************BISECT STEP 5***************** Instructions: 13,031,027,292 Status: slow commit *****************BISECT STEP 6***************** Instructions: 12,711,748,738 Status: fast commit *****************BISECT STEP 7***************** Instructions: 12,711,748,788 Status: fast commit *****************BISECT STEP 8***************** Instructions: 13,031,100,493 Status: slow commit *****************BISECT STEP 9***************** Instructions: 12,714,472,954 Status: fast commit ****************BISECT STEP 10***************** Instructions: 12,715,409,153 Status: fast commit ****************BISECT STEP 11***************** Instructions: 12,715,394,739 Status: fast commit
*****************BISECT RESULT*****************
commit 0673ecdf6cb2b1445a85283db8cbacb251c46516
Author: Richard Henderson <[email protected]> Date: Tue May 5 10:40:23 2020 -0700
softfloat: Inline float64 compare specializations
Replace the float64 compare specializations with inline functions that call the standard float64_compare{,_quiet} functions. Use bool as the return type.
***********************************************
--- scripts/performance/bisect.py | 425 ++++++++++++++++++++++++++++++++++ 1 file changed, 425 insertions(+) create mode 100755 scripts/performance/bisect.py
scripts/performance/bisect.py new file +425 -0
@@ -0,0 +1,425 @@ +#!/usr/bin/env python3 + +""" +Locate the commit that caused a performance degradation or improvement in +QEMU using the git bisect command (binary search). + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" + +import argparse +import multiprocessing +import tempfile +import os +import shutil +import subprocess +import sys + +from typing import List + + +# --------------------------- GIT WRAPPERS -------------------------- +def git_bisect(qemu_path: str, qemu_build_path: str, command: str, + args: List[str] = None) -> str: + """ + Wrapper function for running git bisect. + + Parameters: + qemu_path (str): QEMU path + qemu_build_path (str): Path to the build directory with configuration files + command (str): bisect command (start|fast|slow|reset) + args (list): Optional arguments + + Returns: + (str): git bisect stdout. + """ + process = ["git", "bisect", command] + if args: + process += args + bisect = subprocess.run(process, + cwd=qemu_path, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False) + if bisect.returncode: + clean_exit(qemu_build_path, bisect.stderr.decode("utf-8")) + + return bisect.stdout.decode("utf-8") + + +def git_checkout(commit: str, qemu_path: str, qemu_build_path: str) -> None: + """ + Wrapper function for checking out a given git commit. + + Parameters: + commit (str): Commit hash of a git commit + qemu_path (str): QEMU path + qemu_build_path (str): Path to the build directory with configuration files + """ + checkout_commit = subprocess.run(["git", + "checkout", + commit], + cwd=qemu_path, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if checkout_commit.returncode: + clean_exit(qemu_build_path, checkout_commit.stderr.decode("utf-8")) + + +def git_clone(qemu_path: str) -> None: + """ + Wrapper function for cloning QEMU git repo from GitHub. + + Parameters: + qemu_path (str): Path to clone the QEMU repo to + """ + clone_qemu = subprocess.run(["git", + "clone", + "https://github.com/qemu/qemu.git", + qemu_path], + stderr=subprocess.STDOUT, + check=False) + if clone_qemu.returncode: + sys.exit("Failed to clone QEMU!") +# ------------------------------------------------------------------- + + +def check_requirements(tool: str) -> None: + """ + Verify that all script requirements are installed (perf|callgrind & git). + + Parameters: + tool (str): Tool used for the measurement (perf or callgrind) + """ + if tool == "perf": + check_perf_installation = subprocess.run(["which", "perf"], + stdout=subprocess.DEVNULL, + check=False) + if check_perf_installation.returncode: + sys.exit("Please install perf before running the script.") + + # Insure user has previllage to run perf + check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + check=False) + if check_perf_executability.returncode: + sys.exit(""" + Error: + You may not have permission to collect stats. + Consider tweaking /proc/sys/kernel/perf_event_paranoid, + which controls use of the performance events system by + unprivileged users (without CAP_SYS_ADMIN). + -1: Allow use of (almost) all events by all users + Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK + 0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN + Disallow raw tracepoint access by users without CAP_SYS_ADMIN + 1: Disallow CPU event access by users without CAP_SYS_ADMIN + 2: Disallow kernel profiling by users without CAP_SYS_ADMIN + To make this setting permanent, edit /etc/sysctl.conf too, e.g.: + kernel.perf_event_paranoid = -1 + + *Alternatively, you can run this script under sudo privileges. + """) + elif tool == "callgrind": + check_valgrind_installation = subprocess.run(["which", "valgrind"], + stdout=subprocess.DEVNULL, + check=False) + if check_valgrind_installation.returncode: + sys.exit("Please install valgrind before running the script.") + + # Insure that git is installed + check_git_installation = subprocess.run(["which", "git"], + stdout=subprocess.DEVNULL, + check=False) + if check_git_installation.returncode: + sys.exit("Please install git before running the script.") + + +def clean_exit(qemu_build_path: str, error_message: str) -> None: + """ + Clean up intermediate files and exit. + + Parameters: + qemu_build_path (str): Path to the build directory with configuration files + error_message (str): Error message to display after exiting + """ + shutil.rmtree(qemu_build_path) + sys.exit(error_message) + + +def make(qemu_build_path: str) -> None: + """ + Build QEMU by running the Makefile. + + Parameters: + qemu_build_path (str): Path to the build directory with configuration files + """ + run_make = subprocess.run(["make", + "-j", + str(multiprocessing.cpu_count())], + cwd=qemu_build_path, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if run_make.returncode: + clean_exit(qemu_build_path, run_make.stderr.decode("utf-8")) + + +def measure_instructions(tool: str, qemu_build_path: str, target: str, + command: List[str]) -> int: + """ + Measure the number of instructions when running an program with QEMU. + + Parameters: + tool (str): Tool used for the measurement (perf|callgrind) + qemu_build_path (str): Path to the build directory with configuration files + target (str): QEMU target + command (list): Program path and arguments + + Returns: + (int): Number of instructions. + """ + qemu_exe_path = os.path.join(qemu_build_path, + "{}-linux-user".format(target), + "qemu-{}".format(target)) + instructions = 0 + if tool == "perf": + run_perf = subprocess.run((["perf", + "stat", + "-x", + " ", + "-e", + "instructions", + qemu_exe_path] + + command), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if run_perf.returncode: + clean_exit(qemu_build_path, run_perf.stderr.decode("utf-8")) + + else: + perf_output = run_perf.stderr.decode("utf-8").split(" ") + instructions = int(perf_output[0]) + + elif tool == "callgrind": + with tempfile.NamedTemporaryFile() as tmpfile: + run_callgrind = subprocess.run((["valgrind", + "--tool=callgrind", + "--callgrind-out-file={}".format( + tmpfile.name), + qemu_exe_path] + + command), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if run_callgrind.returncode: + clean_exit(qemu_build_path, run_callgrind.stderr.decode("utf-8")) + + else: + callgrind_output = run_callgrind.stderr.decode("utf-8").split("\n") + instructions = int(callgrind_output[8].split(" ")[-1]) + + return instructions + + +def main(): + """ + Parse the command line arguments then start the execution. + + Syntax: + bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \ + --target TARGET --tool {perf,callgrind} -- \ + <target executable> [<target executable options>] + + Arguments: + [-h] - Print the script arguments help message + -s,--start START - First commit hash in the search range + [-e,--end END] - Last commit hash in the search range + (default: Latest commit) + [-q,--qemu QEMU] - QEMU path. + (default: Path to a GitHub QEMU clone) + --target TARGET - QEMU target name + --tool {perf,callgrind} - Underlying tool used for measurements + + Example of usage: + bisect.py --start=fdd76fecdd --qemu=/path/to/qemu --target=ppc \ + --tool=perf coulomb_double-ppc -n 1000 + """ + + # Parse the command line arguments + parser = argparse.ArgumentParser( + usage="bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU]" + " --target TARGET --tool {perf,callgrind} -- " + "<target executable> [<target executable options>]") + + parser.add_argument("-s", "--start", dest="start", type=str, required=True, + help="First commit hash in the search range") + parser.add_argument("-e", "--end", dest="end", type=str, default="master", + help="Last commit hash in the search range") + parser.add_argument("-q", "--qemu", dest="qemu", type=str, default="", + help="QEMU path") + parser.add_argument("--target", dest="target", type=str, required=True, + help="QEMU target") + parser.add_argument("--tool", dest="tool", choices=["perf", "callgrind"], + required=True, help="Tool used for measurements") + + parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS) + + args = parser.parse_args() + + # Extract the needed variables from the args + start_commit = args.start + end_commit = args.end + qemu = args.qemu + target = args.target + tool = args.tool + command = args.command + + # Set QEMU path + if qemu == "": + # Create a temp directory for cloning QEMU + tmpdir = tempfile.TemporaryDirectory() + qemu_path = os.path.join(tmpdir.name, "qemu") + + # Clone QEMU into the temporary directory + print("Fetching QEMU: ", end="", flush=True) + git_clone(qemu_path) + print() + else: + qemu_path = qemu + + # Create the build directory + qemu_build_path = os.path.join(qemu_path, "tmp-build-gcc") + + if not os.path.exists(qemu_build_path): + os.mkdir(qemu_build_path) + else: + sys.exit("A build directory with the same name (tmp-build-gcc) used in" + " the script is already in the provided QEMU path.") + + # Configure QEMU + configure = subprocess.run(["../configure", + "--target-list={}-linux-user".format(target)], + cwd=qemu_build_path, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if configure.returncode: + clean_exit(qemu_build_path, configure.stderr.decode("utf-8")) + + # Do performance measurements for the start commit + git_checkout(start_commit, qemu_path, qemu_build_path) + make(qemu_build_path) + start_commit_instructions = measure_instructions(tool, + qemu_build_path, + target, + command) + print("{:<30} {}".format("Start Commit Instructions:", + format(start_commit_instructions, ","))) + + # Do performance measurements for the end commit + git_checkout(end_commit, qemu_path, qemu_build_path) + make(qemu_build_path) + end_commit_instructions = measure_instructions(tool, + qemu_build_path, + target, + command) + print("{:<30} {}".format("End Commit Instructions:", + format(end_commit_instructions, ","))) + + # Calculate performance difference between start and end commits + performance_difference = \ + (start_commit_instructions - end_commit_instructions) / \ + max(end_commit_instructions, start_commit_instructions) * 100 + performance_change = "+" if performance_difference > 0 else "-" + print("{:<30} {}".format("Performance Change:", + performance_change + + str(round(abs(performance_difference), 3))+"%")) + + # Set the custom terms used for progressing in "git bisect" + term_old = "fast" if performance_difference < 0 else "slow" + term_new = "slow" if term_old == "fast" else "fast" + + # Start git bisect + git_bisect(qemu_path, qemu_build_path, "start", + ["--term-old", term_old, "--term-new", term_new]) + # Set start commit state + git_bisect(qemu_path, qemu_build_path, term_old, [start_commit]) + # Set end commit state + bisect_output = git_bisect( + qemu_path, qemu_build_path, term_new, [end_commit]) + # Print estimated bisect steps + print("\n{:<30} {}\n".format( + "Estimated Number of Steps:", bisect_output.split()[9])) + + # Initialize bisect_count to track the number of performed + bisect_count = 1 + + while True: + print("**************BISECT STEP {}**************". + format(bisect_count)) + + make(qemu_build_path) + + instructions = measure_instructions(tool, + qemu_build_path, + target, + command) + # Find the difference between the current instructions and start/end + # instructions. + diff_end = abs(instructions - end_commit_instructions) + diff_start = abs(instructions - start_commit_instructions) + + # If current number of insructions is closer to that of start, + # set current commit as term_old. + # Else, set current commit as term_new. + if diff_end > diff_start: + bisect_command = term_old + else: + bisect_command = term_new + + print("{:<20} {}".format("Instructions:", format(instructions, ","))) + print("{:<20} {}".format("Status:", "{} commit". + format(bisect_command))) + + bisect_output = git_bisect(qemu_path, qemu_build_path, bisect_command) + + # Continue if still bisecting, + # else, print result and break. + if not bisect_output.split(" ")[0] == "Bisecting:": + print("\n*****************BISECT RESULT*****************") + commit_message_start = bisect_output.find("commit\n") + 7 + commit_message_end = bisect_output.find(":040000") - 1 + print(bisect_output[commit_message_start:commit_message_end]) + break + + bisect_count += 1 + + # Reset git bisect + git_bisect(qemu_path, qemu_build_path, "reset") + + # Delete temp build directory + shutil.rmtree(qemu_build_path) + + +if __name__ == "__main__": + main()
Signature · 1 line
2.17.1
PATCH 7/9 tests/performance: Add nightly tests
In reply to Ahmed KaramanA nightly performance testing system to monitor any change in QEMU performance across seventeen different targets.
The system includes eight different benchmarks to provide a variety of testing workloads.
dijkstra_double:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are double values. The value of n can be specified using the -n flag. The default value is 2000.
dijkstra_int32:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are int32 values. The value of n can be specified using the -n flag. The default value is 2000.
matmult_double:
Standard matrix multiplication of an n*n matrix of randomly generated double numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
matmult_int32:
Standard matrix multiplication of an n*n matrix of randomly generated integer numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
qsort_double:
Quick sort of an array of n randomly generated double numbers from 0 to 1000. The value of n is passed as an argument with the -n flag. The default value is 300000.
qsort_int32:
Quick sort of an array of n randomly generated integer numbers from 0 to 50000000. The value of n is passed as an argument with the -n flag.The default value is 300000.
qsort_string:
Quick sort of an array of 10000 randomly generated strings of size 8 (including null terminating character). The sort process is repeated n number of times. The value of n is passed as an argument with the -n flag. The default value is 20.
search_string:
Search for the occurrence of a small string in a much larger random string (“needle in a hay”). The search process is repeated n number of times and each time, a different large random string (“hay”) is generated. The value of n can be specified using the -n flag. The default value is 20.
Syntax:
nightly_tests_core.py [-h] [-r REF]
Optional arguments:
-h, --help Show this help message and exit
-r REF, --reference REF
Reference QEMU version - Default is v5.1.0
Example of usage:
nightly_tests_core.py -r v5.1.0 2>log.txt
The following report includes detailed setup and execution details of the system:
https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/
--- tests/performance/nightly-tests/README.md | 243 +++++ .../source/dijkstra_double/dijkstra_double.c | 194 ++++ .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ .../source/matmult_double/matmult_double.c | 123 +++ .../source/matmult_int32/matmult_int32.c | 121 +++ .../source/qsort_double/qsort_double.c | 104 ++ .../source/qsort_int32/qsort_int32.c | 103 ++ .../source/qsort_string/qsort_string.c | 122 +++ .../source/search_string/search_string.c | 110 +++ .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ .../scripts/run_nightly_tests.py | 135 +++ .../nightly-tests/scripts/send_email.py | 56 ++ 12 files changed, 2423 insertions(+) create mode 100644 tests/performance/nightly-tests/README.md create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py create mode 100644 tests/performance/nightly-tests/scripts/send_email.py
tests/performance/nightly-tests/README.md new file +243 -0
@@ -0,0 +1,243 @@ +### QEMU Nightly Tests + +**Required settings:** + +Update the `GMAIL_USER` object in `send_email.py` with your credentials. + +For more details on how the system works, please check the [eighth report](https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/) of the "TCG Continuos Benchmarking" series. + +**Running the System:** + +The default reference version is v5.1.0. To specify a custom version, please use the `-r, --reference` flag. + +```bash +./run_nightly_tests.py +``` + +**Output:** + +``` +Host CPU : Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz +Host Memory : 15.49 GB + +Start Time (UTC) : 2020-08-25 21:30:01 +End Time (UTC) : 2020-08-25 22:02:37 +Execution Time : 0:32:35.896990 + +Status : SUCCESS + +Note: +Changes denoted by '-----' are less than 0.01%. + +-------------------------------------------------------- + SUMMARY REPORT - COMMIT d1a2b51f +-------------------------------------------------------- + AVERAGE RESULTS +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 158 355 274 ----- +1.693% +alpha 1 914 967 171 ----- +3.524% +arm 8 076 402 940 ----- +2.304% +hppa 4 261 685 987 -0.182% +3.164% +m68k 2 690 273 044 ----- +7.131% +mips 1 862 033 667 ----- +2.494% +mipsel 2 008 211 069 ----- +2.674% +mips64 1 918 635 565 ----- +2.818% +mips64el 2 051 565 677 ----- +3.026% +ppc 2 480 141 217 ----- +3.107% +ppc64 2 576 713 959 ----- +3.143% +ppc64le 2 558 853 539 ----- +3.173% +riscv64 1 406 704 050 ----- +2.65% +s390x 3 158 140 046 ----- +3.118% +sh4 2 364 449 748 ----- +3.33% +sparc64 3 318 544 783 ----- +3.851% +x86_64 1 775 844 158 ----- +2.156% +-------------------------------------------------------- + + DETAILED RESULTS +-------------------------------------------------------- +Test Program: dijkstra_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 3 062 583 464 ----- +1.424% +alpha 3 191 864 698 ----- +3.696% +arm 16 357 157 526 ----- +2.347% +hppa 7 228 376 315 -0.139% +3.086% +m68k 4 294 016 587 ----- +9.692% +mips 3 051 419 166 ----- +2.427% +mipsel 3 231 509 618 ----- +2.869% +mips64 3 245 837 754 ----- +2.596% +mips64el 3 414 195 796 ----- +3.021% +ppc 4 914 520 972 -0.041% +4.74% +ppc64 5 098 154 311 ----- +4.565% +ppc64le 5 082 419 054 ----- +4.58% +riscv64 2 192 294 915 ----- +1.955% +s390x 4 584 503 977 ----- +2.896% +sh4 3 949 036 447 ----- +3.464% +sparc64 4 586 203 546 ----- +4.237% +x86_64 2 484 092 105 ----- +1.75% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: dijkstra_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 210 194 577 ----- +1.493% +alpha 1 494 133 274 ----- +2.15% +arm 8 262 935 967 ----- +2.665% +hppa 5 207 318 306 ----- +3.047% +m68k 1 725 856 962 ----- +2.527% +mips 1 495 227 032 ----- +1.492% +mipsel 1 497 147 869 ----- +1.479% +mips64 1 715 388 570 ----- +1.892% +mips64el 1 695 276 864 ----- +1.913% +ppc 2 014 557 389 ----- +1.819% +ppc64 2 206 267 901 ----- +2.139% +ppc64le 2 197 998 781 ----- +2.146% +riscv64 1 354 912 745 ----- +2.396% +s390x 2 916 247 062 ----- +1.241% +sh4 1 990 532 533 ----- +2.669% +sparc64 2 872 231 051 ----- +3.758% +x86_64 1 553 981 241 ----- +2.12% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: matmult_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 1 412 273 223 ----- +0.302% +alpha 3 233 991 649 ----- +7.473% +arm 8 545 173 979 ----- +1.088% +hppa 3 483 597 802 -1.267% +4.468% +m68k 3 919 065 529 ----- +18.431% +mips 2 344 774 894 ----- +4.091% +mipsel 3 329 886 464 ----- +5.177% +mips64 2 359 046 988 ----- +4.076% +mips64el 3 343 664 785 ----- +5.167% +ppc 3 209 457 051 ----- +3.246% +ppc64 3 287 503 981 ----- +3.173% +ppc64le 3 287 189 065 ----- +3.173% +riscv64 1 221 603 682 ----- +0.277% +s390x 2 874 199 923 ----- +5.827% +sh4 3 543 943 634 ----- +6.416% +sparc64 3 426 150 004 ----- +7.139% +x86_64 1 248 917 276 ----- +0.322% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: matmult_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 598 681 621 ----- +0.585% +alpha 372 437 418 ----- +0.677% +arm 746 583 193 ----- +1.462% +hppa 674 278 359 ----- +1.183% +m68k 410 495 553 ----- +0.9% +mips 499 698 837 ----- +0.531% +mipsel 499 500 429 ----- +0.497% +mips64 481 554 664 ----- +0.599% +mips64el 465 057 054 ----- +0.619% +ppc 341 334 603 ----- +0.944% +ppc64 393 796 203 ----- +0.966% +ppc64le 393 977 298 ----- +0.965% +riscv64 351 709 769 ----- +0.785% +s390x 494 427 384 ----- +0.599% +sh4 402 668 444 ----- +0.899% +sparc64 495 952 959 ----- +1.192% +x86_64 402 928 461 ----- +0.833% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 709 683 624 ----- +2.417% +alpha 1 969 460 172 ----- +3.68% +arm 8 322 998 390 ----- +2.587% +hppa 3 188 301 995 -0.047% +2.9% +m68k 4 953 930 065 ----- +15.153% +mips 2 123 919 587 ----- +3.055% +mipsel 2 124 212 187 ----- +3.048% +mips64 1 999 047 826 ----- +3.405% +mips64el 1 996 426 772 ----- +3.409% +ppc 2 819 267 902 -0.021% +5.435% +ppc64 2 768 186 548 ----- +5.513% +ppc64le 2 724 803 772 -0.011% +5.603% +riscv64 1 638 328 937 ----- +4.021% +s390x 2 519 081 708 ----- +3.362% +sh4 2 595 545 154 ----- +2.994% +sparc64 3 988 986 587 ----- +2.747% +x86_64 2 033 468 588 ----- +3.234% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 193 392 565 ----- +2.916% +alpha 1 521 291 933 ----- +4.193% +arm 3 465 445 043 ----- +2.756% +hppa 2 280 034 340 ----- +3.821% +m68k 1 843 189 041 ----- +3.583% +mips 1 558 024 873 ----- +3.863% +mipsel 1 560 583 980 ----- +3.846% +mips64 1 563 415 749 ----- +4.412% +mips64el 1 542 677 320 ----- +4.474% +ppc 1 728 698 880 ----- +3.665% +ppc64 1 842 444 545 ----- +3.555% +ppc64le 1 791 822 067 ----- +3.661% +riscv64 1 348 866 430 ----- +4.654% +s390x 2 184 073 151 ----- +3.319% +sh4 1 946 492 539 ----- +3.624% +sparc64 3 452 215 585 ----- +2.937% +x86_64 1 813 544 414 ----- +3.537% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_string +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 592 218 418 ----- +2.468% +alpha 1 855 834 626 ----- +3.487% +arm 7 347 721 165 ----- +2.682% +hppa 4 758 753 926 ----- +3.543% +m68k 2 376 811 462 ----- +3.567% +mips 2 166 608 045 ----- +2.532% +mipsel 2 163 392 541 ----- +2.528% +mips64 2 029 251 969 ----- +3.117% +mips64el 2 011 628 621 ----- +3.145% +ppc 2 492 942 463 ----- +2.673% +ppc64 2 464 702 554 ----- +2.488% +ppc64le 2 445 253 307 ----- +2.505% +riscv64 1 625 053 328 ----- +3.953% +s390x 4 194 608 798 ----- +6.623% +sh4 2 164 142 539 ----- +3.166% +sparc64 4 299 516 539 ----- +4.065% +x86_64 2 940 456 780 ----- +2.649% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: search_string +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 487 814 704 ----- +1.94% +alpha 1 680 723 605 ----- +2.835% +arm 11 563 208 260 ----- +2.848% +hppa 7 272 826 858 ----- +3.263% +m68k 1 998 819 159 ----- +3.198% +mips 1 656 596 909 ----- +1.959% +mipsel 1 659 455 464 ----- +1.947% +mips64 1 955 541 001 ----- +2.447% +mips64el 1 943 598 207 ----- +2.462% +ppc 2 320 350 477 ----- +2.332% +ppc64 2 552 655 634 ----- +2.742% +ppc64le 2 547 364 971 ----- +2.748% +riscv64 1 520 862 601 ----- +3.159% +s390x 5 497 978 370 ----- +1.078% +sh4 2 323 236 696 ----- +3.41% +sparc64 3 427 101 999 ----- +4.73% +x86_64 1 729 364 402 ----- +2.806% +-------------------------------------------------------- +```
tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c new file +194 -0
194 lines not on this page · raw message
tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c new file +192 -0
192 lines not on this page · raw message
tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c new file +123 -0
123 lines not on this page · raw message
tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c new file +121 -0
121 lines not on this page · raw message
tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c new file +104 -0
104 lines not on this page · raw message
tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c new file +103 -0
103 lines not on this page · raw message
tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c new file +122 -0
122 lines not on this page · raw message
tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c new file +110 -0
110 lines not on this page · raw message
tests/performance/nightly-tests/scripts/nightly_tests_core.py new file +920 -0
920 lines not on this page · raw message
tests/performance/nightly-tests/scripts/run_nightly_tests.py new file +135 -0
135 lines not on this page · raw message
tests/performance/nightly-tests/scripts/send_email.py new file +56 -0
56 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH 7/9] tests/performance: Add nightly tests
Quoted 4 lines · Ahmed Karaman
A nightly performance testing system to monitor any change in QEMU performance across seventeen different targets.
Quoted 1,248 lines
The system includes eight different benchmarks to provide a variety of testing workloads.
dijkstra_double:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are double values. The value of n can be specified using the -n flag. The default value is 2000.
dijkstra_int32:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are int32 values. The value of n can be specified using the -n flag. The default value is 2000.
matmult_double:
Standard matrix multiplication of an n*n matrix of randomly generated double numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
matmult_int32:
Standard matrix multiplication of an n*n matrix of randomly generated integer numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
qsort_double:
Quick sort of an array of n randomly generated double numbers from 0 to 1000. The value of n is passed as an argument with the -n flag. The default value is 300000.
qsort_int32:
Quick sort of an array of n randomly generated integer numbers from 0 to 50000000. The value of n is passed as an argument with the -n flag.The default value is 300000.
qsort_string:
Quick sort of an array of 10000 randomly generated strings of size 8 (including null terminating character). The sort process is repeated n number of times. The value of n is passed as an argument with the -n flag. The default value is 20.
search_string:
Search for the occurrence of a small string in a much larger random string (“needle in a hay”). The search process is repeated n number of times and each time, a different large random string (“hay”) is generated. The value of n can be specified using the -n flag. The default value is 20.Syntax:
nightly_tests_core.py[-h] [-r REF]
Optional arguments:
-h, --help Show this help message and exit
-r REF, --reference REF
Reference QEMU version - Default is v5.1.0
Example of usage:
nightly_tests_core.py-r v5.1.0 2>log.txtThe following report includes detailed setup and execution details of the system:
https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/
QEMU-Nightly-Performance-Tests/Signed-off-by Ahmed Karaman <[email protected]>--- tests/performance/nightly-tests/README.md | 243 +++++ .../source/dijkstra_double/dijkstra_double.c | 194 ++++ .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ .../source/matmult_double/matmult_double.c | 123 +++ .../source/matmult_int32/matmult_int32.c | 121 +++ .../source/qsort_double/qsort_double.c | 104 ++ .../source/qsort_int32/qsort_int32.c | 103 ++ .../source/qsort_string/qsort_string.c | 122 +++ .../source/search_string/search_string.c | 110 +++ .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ .../scripts/run_nightly_tests.py | 135 +++ .../nightly-tests/scripts/send_email.py | 56 ++ 12 files changed, 2423 insertions(+) create mode 100644 tests/performance/nightly-tests/README.md create mode 100644 tests/performance/nightly-tests/benchmarks/source/ dijkstra_double/dijkstra_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/ dijkstra_int32/dijkstra_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/ matmult_double/matmult_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/ matmult_int32/matmult_int32.c create mode 100644 tests/performance/nightly- tests/benchmarks/source/qsort_double/qsort_double.c create mode 100644 tests/performance/nightly- tests/benchmarks/source/qsort_int32/qsort_int32.c create mode 100644 tests/performance/nightly- tests/benchmarks/source/qsort_string/qsort_string.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/ search_string/search_string.c create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_ core.py create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_ tests.py create mode 100644 tests/performance/nightly-tests/scripts/send_email.pytests/performance/nightly-tests/README.md new file +7
-0@@ -0,0 +1,243 @@ +### QEMU Nightly Tests + +**Required settings:** + +Update the `GMAIL_USER` object in `send_email.py` with your credentials. + +For more details on how the system works, please check the [eighthreport](https://ahmedkrmn.github.io/TCG-Continuous- Benchmarking/QEMU-Nightly-Performance-Tests/) of the "TCG Continuos Benchmarking" series.
tests/performance/nightly-tests/README.md new file +4
-0+ +**Running the System:** + +The default reference version is v5.1.0. To specify a custom version,please use the
-r, --referenceflag.tests/performance/nightly-tests/README.md new file +232
-0+ +```bash +./run_nightly_tests.py +``` + +**Output:** + +``` +Host CPU : Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz +Host Memory : 15.49 GB + +Start Time (UTC) : 2020-08-25 21:30:01 +End Time (UTC) : 2020-08-25 22:02:37 +Execution Time : 0:32:35.896990 + +Status : SUCCESS + +Note: +Changes denoted by '-----' are less than 0.01%. + +-------------------------------------------------------- + SUMMARY REPORT - COMMIT d1a2b51f +-------------------------------------------------------- + AVERAGE RESULTS +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 158 355 274 ----- +1.693% +alpha 1 914 967 171 ----- +3.524% +arm 8 076 402 940 ----- +2.304% +hppa 4 261 685 987 -0.182% +3.164% +m68k 2 690 273 044 ----- +7.131% +mips 1 862 033 667 ----- +2.494% +mipsel 2 008 211 069 ----- +2.674% +mips64 1 918 635 565 ----- +2.818% +mips64el 2 051 565 677 ----- +3.026% +ppc 2 480 141 217 ----- +3.107% +ppc64 2 576 713 959 ----- +3.143% +ppc64le 2 558 853 539 ----- +3.173% +riscv64 1 406 704 050 ----- +2.65% +s390x 3 158 140 046 ----- +3.118% +sh4 2 364 449 748 ----- +3.33% +sparc64 3 318 544 783 ----- +3.851% +x86_64 1 775 844 158 ----- +2.156% +-------------------------------------------------------- + + DETAILED RESULTS +-------------------------------------------------------- +Test Program: dijkstra_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 3 062 583 464 ----- +1.424% +alpha 3 191 864 698 ----- +3.696% +arm 16 357 157 526 ----- +2.347% +hppa 7 228 376 315 -0.139% +3.086% +m68k 4 294 016 587 ----- +9.692% +mips 3 051 419 166 ----- +2.427% +mipsel 3 231 509 618 ----- +2.869% +mips64 3 245 837 754 ----- +2.596% +mips64el 3 414 195 796 ----- +3.021% +ppc 4 914 520 972 -0.041% +4.74% +ppc64 5 098 154 311 ----- +4.565% +ppc64le 5 082 419 054 ----- +4.58% +riscv64 2 192 294 915 ----- +1.955% +s390x 4 584 503 977 ----- +2.896% +sh4 3 949 036 447 ----- +3.464% +sparc64 4 586 203 546 ----- +4.237% +x86_64 2 484 092 105 ----- +1.75% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: dijkstra_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 210 194 577 ----- +1.493% +alpha 1 494 133 274 ----- +2.15% +arm 8 262 935 967 ----- +2.665% +hppa 5 207 318 306 ----- +3.047% +m68k 1 725 856 962 ----- +2.527% +mips 1 495 227 032 ----- +1.492% +mipsel 1 497 147 869 ----- +1.479% +mips64 1 715 388 570 ----- +1.892% +mips64el 1 695 276 864 ----- +1.913% +ppc 2 014 557 389 ----- +1.819% +ppc64 2 206 267 901 ----- +2.139% +ppc64le 2 197 998 781 ----- +2.146% +riscv64 1 354 912 745 ----- +2.396% +s390x 2 916 247 062 ----- +1.241% +sh4 1 990 532 533 ----- +2.669% +sparc64 2 872 231 051 ----- +3.758% +x86_64 1 553 981 241 ----- +2.12% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: matmult_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 1 412 273 223 ----- +0.302% +alpha 3 233 991 649 ----- +7.473% +arm 8 545 173 979 ----- +1.088% +hppa 3 483 597 802 -1.267% +4.468% +m68k 3 919 065 529 ----- +18.431% +mips 2 344 774 894 ----- +4.091% +mipsel 3 329 886 464 ----- +5.177% +mips64 2 359 046 988 ----- +4.076% +mips64el 3 343 664 785 ----- +5.167% +ppc 3 209 457 051 ----- +3.246% +ppc64 3 287 503 981 ----- +3.173% +ppc64le 3 287 189 065 ----- +3.173% +riscv64 1 221 603 682 ----- +0.277% +s390x 2 874 199 923 ----- +5.827% +sh4 3 543 943 634 ----- +6.416% +sparc64 3 426 150 004 ----- +7.139% +x86_64 1 248 917 276 ----- +0.322% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: matmult_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 598 681 621 ----- +0.585% +alpha 372 437 418 ----- +0.677% +arm 746 583 193 ----- +1.462% +hppa 674 278 359 ----- +1.183% +m68k 410 495 553 ----- +0.9% +mips 499 698 837 ----- +0.531% +mipsel 499 500 429 ----- +0.497% +mips64 481 554 664 ----- +0.599% +mips64el 465 057 054 ----- +0.619% +ppc 341 334 603 ----- +0.944% +ppc64 393 796 203 ----- +0.966% +ppc64le 393 977 298 ----- +0.965% +riscv64 351 709 769 ----- +0.785% +s390x 494 427 384 ----- +0.599% +sh4 402 668 444 ----- +0.899% +sparc64 495 952 959 ----- +1.192% +x86_64 402 928 461 ----- +0.833% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 709 683 624 ----- +2.417% +alpha 1 969 460 172 ----- +3.68% +arm 8 322 998 390 ----- +2.587% +hppa 3 188 301 995 -0.047% +2.9% +m68k 4 953 930 065 ----- +15.153% +mips 2 123 919 587 ----- +3.055% +mipsel 2 124 212 187 ----- +3.048% +mips64 1 999 047 826 ----- +3.405% +mips64el 1 996 426 772 ----- +3.409% +ppc 2 819 267 902 -0.021% +5.435% +ppc64 2 768 186 548 ----- +5.513% +ppc64le 2 724 803 772 -0.011% +5.603% +riscv64 1 638 328 937 ----- +4.021% +s390x 2 519 081 708 ----- +3.362% +sh4 2 595 545 154 ----- +2.994% +sparc64 3 988 986 587 ----- +2.747% +x86_64 2 033 468 588 ----- +3.234% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 193 392 565 ----- +2.916% +alpha 1 521 291 933 ----- +4.193% +arm 3 465 445 043 ----- +2.756% +hppa 2 280 034 340 ----- +3.821% +m68k 1 843 189 041 ----- +3.583% +mips 1 558 024 873 ----- +3.863% +mipsel 1 560 583 980 ----- +3.846% +mips64 1 563 415 749 ----- +4.412% +mips64el 1 542 677 320 ----- +4.474% +ppc 1 728 698 880 ----- +3.665% +ppc64 1 842 444 545 ----- +3.555% +ppc64le 1 791 822 067 ----- +3.661% +riscv64 1 348 866 430 ----- +4.654% +s390x 2 184 073 151 ----- +3.319% +sh4 1 946 492 539 ----- +3.624% +sparc64 3 452 215 585 ----- +2.937% +x86_64 1 813 544 414 ----- +3.537% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_string +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 592 218 418 ----- +2.468% +alpha 1 855 834 626 ----- +3.487% +arm 7 347 721 165 ----- +2.682% +hppa 4 758 753 926 ----- +3.543% +m68k 2 376 811 462 ----- +3.567% +mips 2 166 608 045 ----- +2.532% +mipsel 2 163 392 541 ----- +2.528% +mips64 2 029 251 969 ----- +3.117% +mips64el 2 011 628 621 ----- +3.145% +ppc 2 492 942 463 ----- +2.673% +ppc64 2 464 702 554 ----- +2.488% +ppc64le 2 445 253 307 ----- +2.505% +riscv64 1 625 053 328 ----- +3.953% +s390x 4 194 608 798 ----- +6.623% +sh4 2 164 142 539 ----- +3.166% +sparc64 4 299 516 539 ----- +4.065% +x86_64 2 940 456 780 ----- +2.649% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: search_string +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 487 814 704 ----- +1.94% +alpha 1 680 723 605 ----- +2.835% +arm 11 563 208 260 ----- +2.848% +hppa 7 272 826 858 ----- +3.263% +m68k 1 998 819 159 ----- +3.198% +mips 1 656 596 909 ----- +1.959% +mipsel 1 659 455 464 ----- +1.947% +mips64 1 955 541 001 ----- +2.447% +mips64el 1 943 598 207 ----- +2.462% +ppc 2 320 350 477 ----- +2.332% +ppc64 2 552 655 634 ----- +2.742% +ppc64le 2 547 364 971 ----- +2.748% +riscv64 1 520 862 601 ----- +3.159% +s390x 5 497 978 370 ----- +1.078% +sh4 2 323 236 696 ----- +3.41% +sparc64 3 427 101 999 ----- +4.73% +x86_64 1 729 364 402 ----- +2.806% +-------------------------------------------------------- +```(quoted diff) +0
-0
dijkstra_double/dijkstra_double.cb/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.cnew file mode 100644
index 0000000000..9c0bb804actests/performance/nightly-tests/benchmarks/source/ new file +0
-0
dijkstra_double/dijkstra_double.ctests/performance/nightly-tests/benchmarks/source/ new file +11
-0@@ -0,0 +1,194 @@ +/* + * Source file of a benchmark program involving calculations of the + * shortest distances between a source node and all other nodes in a + * graph of n nodes in which all nxn distances are defined as "double". + * The number n can be given via command line, and the default is 2000. + * The algorithm used is Dijsktra's. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman <[email protected]> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details.+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */tests/performance/nightly-tests/benchmarks/source/ new file +31
-0+ +#include <float.h> +#include <limits.h> +#include <stdbool.h> +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_NODE_COUNT 2000 +#define MIN_NODE_COUNT 3 +#define MAX_NODE_COUNT 10000 + + +int32_t closest_index(int32_t count, double *distances, bool *flags) +{ + int32_t closest; + double minimum = DBL_MAX; + + for (size_t i = 0; i < count; i++) { + if (flags[i] == false && distances[i] <= minimum) { + closest = i; + minimum = distances[i]; + } + } + + return closest; +} + +/** + * Calculate the shortest distances from the source node using Dijkstramethod.
+ * @param (out) distances An array of shortest distances from the source node.
+ * @param (out) via An array of nodes needed to be taken as the the last
+ * before destination, for each destination.
+ * @param (out) eccent Eccentricity of the source node.
+ * @param (in) count The number of nodes.
+ * @param (in) source Source node.
+ * @param (in) matrix Distance matrix.
+ */
+void find_shortest_distances(double *distances,int32_t*via, double *eccent,
+int32_tcount,int32_tsource, double
**matrix)tests/performance/nightly-tests/benchmarks/source/ new file +21
-0+{ + bool *flags; + + flags = (bool *)malloc(count * sizeof(bool)); + + for (size_t i = 0; i < count; i++) { + distances[i] = DBL_MAX; + flags[i] = false; + } + + distances[source] = 0.0; + via[source] = source; + + for (size_t i = 0; i < count - 1; i++) { + int32_t closest = closest_index(count, distances, flags); + flags[closest] = true; + for (size_t j = 0; j < count; j++) { + if ((!flags[j]) && + (matrix[closest][j]) && + (distances[closest] != DBL_MAX) && + (distances[j] > distances[closest] +matrix[closest][j])) {
tests/performance/nightly-tests/benchmarks/source/ new file +36
-0+ distances[j] = distances[closest] + matrix[closest][j]; + via[j] = closest; + } + } + } + + *eccent = 0; + for (size_t i = 0; i < count; i++) { + if (*eccent < distances[i]) { + *eccent = distances[i]; + } + } + + free(flags); +} + + +void main(int argc, char *argv[]) +{ + double **distance_matrix; + double *shortest_distances; + int32_t *via_node; + int32_t node_count = DEFAULT_NODE_COUNT; + int32_t source_node = 0; + double node_eccentricity = 0.0; + double range_factor = 999.0 / (double)(RAND_MAX); + int32_t option; + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_node_count = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_node_count == 0) { + fprintf(stderr, "Error ... Invalid value for option'-n'.\n");
tests/performance/nightly-tests/benchmarks/source/ new file +5
-0+ exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_node_count < MIN_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe a "
tests/performance/nightly-tests/benchmarks/source/ new file +6
-0+ "number less than %d.\n", MIN_NODE_COUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_node_count > MAX_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe "
tests/performance/nightly-tests/benchmarks/source/ new file +13
-0+ "more than %d.\n", MAX_NODE_COUNT); + exit(EXIT_FAILURE); + } + node_count = user_node_count; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + distance_matrix = (double **)malloc(node_count * sizeof(double *)); + for (size_t i = 0; i < node_count; i++) { + distance_matrix[i] = (double *)malloc(node_count *sizeof(double));
tests/performance/nightly-tests/benchmarks/source/ new file +19
-0+ } + shortest_distances = (double *)malloc(node_count * sizeof(double)); + via_node = (int32_t *)malloc(node_count * sizeof(int32_t)); + + /* Initialize helper arrays and populate distance_matrix */ + srand(1); + for (size_t i = 0; i < node_count; i++) { + shortest_distances[i] = 0.0; + via_node[i] = -1; + distance_matrix[i][i] = 0.0; + } + for (size_t i = 0; i < node_count; i++) { + for (size_t j = i + 1; j < node_count; j++) { + distance_matrix[i][j] = 1.0 + range_factor * (double)rand(); + distance_matrix[j][i] = distance_matrix[i][j]; + } + } + + find_shortest_distances(shortest_distances, via_node,
&node_eccentricity,tests/performance/nightly-tests/benchmarks/source/ new file +27
-0+ node_count, source_node, distance_matrix); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" Distance matrix (top left part):\n"); + for (size_t i = 0; i < 3; i++) { + for (size_t j = 0; j < 3; j++) { + printf(" %7.2f", distance_matrix[i][j]); + } + printf("\n"); + } + printf(" Source: %d (eccentricity: %f)\n", + source_node, node_eccentricity); + printf(" Destination Distance Via Node\n"); + for (size_t i = 0; i < 3; i++) { + printf(" %5d %7.2f %4d\n", + i, shortest_distances[i], via_node[i]); + } + + /* Free all previously allocated space */ + for (size_t i = 0; i < node_count; i++) { + free(distance_matrix[i]); + } + free(distance_matrix); + free(shortest_distances); + free(via_node); +}(quoted diff) +0
-0
dijkstra_int32/dijkstra_int32.cb/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.cnew file mode 100644
index 0000000000..2663cde943tests/performance/nightly-tests/benchmarks/source/ new file +0
-0
dijkstra_int32/dijkstra_int32.ctests/performance/nightly-tests/benchmarks/source/ new file +11
-0@@ -0,0 +1,192 @@ +/* + * Source file of a benchmark program involving calculations of the + * shortest distances between a source node and all other nodes in a + * graph of n nodes in which all nxn distances are defined as "int32". + * The number n can be given via command line, and the default is 2000. + * The algorithm used is Dijsktra's. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman <[email protected]> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details.+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */tests/performance/nightly-tests/benchmarks/source/ new file +30
-0+ +#include <limits.h> +#include <stdbool.h> +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_NODE_COUNT 2000 +#define MIN_NODE_COUNT 3 +#define MAX_NODE_COUNT 10000 + + +int32_t closest_index(int32_t count, int32_t *distances, bool *flags) +{ + int32_t closest; + int32_t minimum = INT_MAX; + + for (size_t i = 0; i < count; i++) { + if (flags[i] == false && distances[i] <= minimum) { + closest = i; + minimum = distances[i]; + } + } + + return closest; +} + +/** + * Calculate the shortest distances from the source node using Dijkstramethod.
+ * @param (out) distances An array of shortest distances from the source node.
+ * @param (out) via An array of nodes needed to be taken as the the last
+ * before destination, for each destination.
+ * @param (out) eccent Eccentricity of the source node.
+ * @param (in) count The number of nodes.
+ * @param (in) source Source node.
+ * @param (in) matrix Distance matrix.
+ */
+void find_shortest_distances(int32_t *distances,int32_t*via,int32_t*eccent,
+int32_tcount,int32_tsource,int32_t
**matrix)tests/performance/nightly-tests/benchmarks/source/ new file +21
-0+{ + bool *flags; + + flags = (bool *)malloc(count * sizeof(bool)); + + for (size_t i = 0; i < count; i++) { + distances[i] = INT_MAX; + flags[i] = false; + } + + distances[source] = 0; + via[source] = source; + + for (size_t i = 0; i < count - 1; i++) { + int32_t closest = closest_index(count, distances, flags); + flags[closest] = true; + for (size_t j = 0; j < count; j++) { + if ((!flags[j]) && + (matrix[closest][j]) && + (distances[closest] != INT_MAX) && + (distances[j] > distances[closest] +matrix[closest][j])) {
tests/performance/nightly-tests/benchmarks/source/ new file +35
-0+ distances[j] = distances[closest] + matrix[closest][j]; + via[j] = closest; + } + } + } + + *eccent = 0; + for (size_t i = 0; i < count; i++) { + if (*eccent < distances[i]) { + *eccent = distances[i]; + } + } + + free(flags); +} + + +void main(int argc, char *argv[]) +{ + int32_t **distance_matrix; + int32_t *shortest_distances; + int32_t *via_node; + int32_t node_count = DEFAULT_NODE_COUNT; + int32_t source_node = 0; + int32_t node_eccentricity = 0; + int32_t option; + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_node_count = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_node_count == 0) { + fprintf(stderr, "Error ... Invalid value for option'-n'.\n");
tests/performance/nightly-tests/benchmarks/source/ new file +5
-0+ exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_node_count < MIN_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe a "
tests/performance/nightly-tests/benchmarks/source/ new file +6
-0+ "number less than %d.\n", MIN_NODE_COUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_node_count > MAX_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe "
tests/performance/nightly-tests/benchmarks/source/ new file +13
-0+ "more than %d.\n", MAX_NODE_COUNT); + exit(EXIT_FAILURE); + } + node_count = user_node_count; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + distance_matrix = (int32_t **)malloc(node_count * sizeof(int32_t *)); + for (size_t i = 0; i < node_count; i++) { + distance_matrix[i] = (int32_t *)malloc(node_count *
sizeof(int32_t));tests/performance/nightly-tests/benchmarks/source/ new file +19
-0+ } + shortest_distances = (int32_t *)malloc(node_count * sizeof(int32_t)); + via_node = (int32_t *)malloc(node_count * sizeof(int32_t)); + + /* Initialize helper arrays and populate distance_matrix */ + srand(1); + for (size_t i = 0; i < node_count; i++) { + shortest_distances[i] = 0; + via_node[i] = -1; + distance_matrix[i][i] = 0; + } + for (size_t i = 0; i < node_count; i++) { + for (size_t j = i + 1; j < node_count; j++) { + distance_matrix[i][j] = 1 + (rand()) / (RAND_MAX / 999); + distance_matrix[j][i] = distance_matrix[i][j]; + } + } + + find_shortest_distances(shortest_distances, via_node,
&node_eccentricity,tests/performance/nightly-tests/benchmarks/source/ new file +27
-0+ node_count, source_node, distance_matrix); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" Distance matrix (top left part):\n"); + for (size_t i = 0; i < 3; i++) { + for (size_t j = 0; j < 3; j++) { + printf(" %6d", distance_matrix[i][j]); + } + printf("\n"); + } + printf(" Source: %d (eccentricity: %d)\n", + source_node, node_eccentricity); + printf(" Destination Distance Via Node\n"); + for (size_t i = 0; i < 3; i++) { + printf(" %5d %3d %4d\n", + i, shortest_distances[i], via_node[i]); + } + + /* Free all previously allocated space */ + for (size_t i = 0; i < node_count; i++) { + free(distance_matrix[i]); + } + free(distance_matrix); + free(shortest_distances); + free(via_node); +}(quoted diff) +0
-0
matmult_double/matmult_double.cb/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.cnew file mode 100644
index 0000000000..42bbb4717atests/performance/nightly-tests/benchmarks/source/ new file +0
-0
matmult_double/matmult_double.ctests/performance/nightly-tests/benchmarks/source/ new file +9
-0@@ -0,0 +1,123 @@ +/* + * Source file of a benchmark program involving calculations of + * a product of two matrixes nxn whose elements are "double". The + * number n can be given via command line, and the default is 200. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman <[email protected]> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details.+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */tests/performance/nightly-tests/benchmarks/source/ new file +31
-0+ +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_MATRIX_SIZE 200 +#define MIN_MATRIX_SIZE 2 +#define MAX_MATRIX_SIZE 200000 + +void main(int argc, char *argv[]) +{ + double **matrix_a; + double **matrix_b; + double **matrix_res; + size_t i; + size_t j; + size_t k; + int32_t matrix_size = DEFAULT_MATRIX_SIZE; + int32_t option; + double range_factor = 100.0 / (double)(RAND_MAX); + + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_matrix_size = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_matrix_size == 0) { + fprintf(stderr, "Error ... Invalid value for option'-n'.\n");
tests/performance/nightly-tests/benchmarks/source/ new file +5
-0+ exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_matrix_size < MIN_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe a "
+ "number less than %d.\n",MIN_MATRIX_SIZE);tests/performance/nightly-tests/benchmarks/source/ new file +5
-0+ exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_matrix_size > MAX_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe "
tests/performance/nightly-tests/benchmarks/source/ new file +57
-0+ "more than %d.\n", MAX_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + matrix_size = user_matrix_size; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + matrix_a = (double **)malloc(matrix_size * sizeof(double *)); + for (i = 0; i < matrix_size; i++) { + matrix_a[i] = (double *)malloc(matrix_size * sizeof(double)); + } + matrix_b = (double **)malloc(matrix_size * sizeof(double *)); + for (i = 0; i < matrix_size; i++) { + matrix_b[i] = (double *)malloc(matrix_size * sizeof(double)); + } + matrix_res = (double **)malloc(matrix_size * sizeof(double *)); + for (i = 0; i < matrix_size; i++) { + matrix_res[i] = (double *)malloc(matrix_size * sizeof(double)); + } + + /* Populate matrix_a and matrix_b with random numbers */ + srand(1); + for (i = 0; i < matrix_size; i++) { + for (j = 0; j < matrix_size; j++) { + matrix_a[i][j] = range_factor * (double)rand(); + matrix_b[i][j] = range_factor * (double)rand(); + } + } + + /* Calculate the product of two matrixes */ + for (i = 0; i < matrix_size; i++) { + for (j = 0; j < matrix_size; j++) { + matrix_res[i][j] = 0.0; + for (k = 0; k < matrix_size; k++) { + matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j]; + } + } + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" %f %f\n", matrix_res[0][0], matrix_res[0][1]); + printf(" %f %f\n", matrix_res[1][0], matrix_res[1][1]); + + /* Free all previously allocated space */ + for (i = 0; i < matrix_size; i++) { + free(matrix_a[i]); + free(matrix_b[i]); + free(matrix_res[i]); + } + free(matrix_a); + free(matrix_b); + free(matrix_res); +}tests/performance/nightly-tests/benchmarks/source/ renamed from tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c +0
-0
matmult_int32/matmult_int32.c
new file mode 100644
index 0000000000..29a6eb000dtests/performance/nightly-tests/benchmarks/source/ new file +0
-0
matmult_int32/matmult_int32.ctests/performance/nightly-tests/benchmarks/source/ new file +9
-0@@ -0,0 +1,121 @@ +/* + * Source file of a benchmark program involving calculations of + * a product of two matrixes nxn whose elements are "int32_t". The + * number n can be given via command line, and the default is 200. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman <[email protected]> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details.+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */tests/performance/nightly-tests/benchmarks/source/ new file +29
-0+ +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_MATRIX_SIZE 200 +#define MIN_MATRIX_SIZE 2 +#define MAX_MATRIX_SIZE 200000 + +void main(int argc, char *argv[]) +{ + int32_t **matrix_a; + int32_t **matrix_b; + int32_t **matrix_res; + size_t i; + size_t j; + size_t k; + int32_t matrix_size = DEFAULT_MATRIX_SIZE; + int32_t option; + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_matrix_size = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_matrix_size == 0) { + fprintf(stderr, "Error ... Invalid value for option'-n'.\n");
tests/performance/nightly-tests/benchmarks/source/ new file +5
-0+ exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_matrix_size < MIN_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe a "
+ "number less than %d.\n",MIN_MATRIX_SIZE);tests/performance/nightly-tests/benchmarks/source/ new file +5
-0+ exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_matrix_size > MAX_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe "
tests/performance/nightly-tests/benchmarks/source/ new file +57
-0+ "more than %d.\n", MAX_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + matrix_size = user_matrix_size; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + matrix_a = (int32_t **)malloc(matrix_size * sizeof(int32_t *)); + for (i = 0; i < matrix_size; i++) { + matrix_a[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t)); + } + matrix_b = (int32_t **)malloc(matrix_size * sizeof(int32_t *)); + for (i = 0; i < matrix_size; i++) { + matrix_b[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t)); + } + matrix_res = (int32_t **)malloc(matrix_size * sizeof(int32_t *)); + for (i = 0; i < matrix_size; i++) { + matrix_res[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t)); + } + + /* Populate matrix_a and matrix_b with random numbers */ + srand(1); + for (i = 0; i < matrix_size; i++) { + for (j = 0; j < matrix_size; j++) { + matrix_a[i][j] = (rand()) / (RAND_MAX / 100); + matrix_b[i][j] = (rand()) / (RAND_MAX / 100); + } + } + + /* Calculate the product of two matrixes */ + for (i = 0; i < matrix_size; i++) { + for (j = 0; j < matrix_size; j++) { + matrix_res[i][j] = 0; + for (k = 0; k < matrix_size; k++) { + matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j]; + } + } + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" %d %d\n", matrix_res[0][0], matrix_res[0][1]); + printf(" %d %d\n", matrix_res[1][0], matrix_res[1][1]); + + /* Free all previously allocated space */ + for (i = 0; i < matrix_size; i++) { + free(matrix_a[i]); + free(matrix_b[i]); + free(matrix_res[i]); + } + free(matrix_a); + free(matrix_b); + free(matrix_res); +}tests/performance/nightly-tests/benchmarks/source/qsort_ renamed from tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c +0
-0
double/qsort_double.c
new file mode 100644
index 0000000000..efc1b2eee1tests/performance/nightly-tests/benchmarks/source/qsort_ new file +0
-0
double/qsort_double.ctests/performance/nightly-tests/benchmarks/source/qsort_ new file +9
-0@@ -0,0 +1,104 @@ +/* + * Source file of a benchmark program involving sorting of an array + * of length n whose elements are "double". The default value for n + * is 300000, and it can be set via command line as well. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman <[email protected]> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details.+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */tests/performance/nightly-tests/benchmarks/source/qsort_ new file +39
-0+ +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> + +/* Number of elements in the array to be sorted */ +#define DEFAULT_ARRAY_LEN 300000 +#define MIN_ARRAY_LEN 3 +#define MAX_ARRAY_LEN 30000000 + +/* Upper limit used for generation of random numbers */ +#define UPPER_LIMIT 1000.0 + +/* Comparison function passed to qsort() */ +static int compare(const void *a, const void *b) +{ + if (*(const double *)a > *(const double *)b) { + return 1; + } else if (*(const double *)a < *(const double *)b) { + return -1; + } + return 0; +} + +void main(int argc, char *argv[]) +{ + double *array_to_be_sorted; + int32_t array_len = DEFAULT_ARRAY_LEN; + int32_t option; + double range_factor = UPPER_LIMIT / (double)(RAND_MAX); + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_array_len = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_array_len == 0) { + fprintf(stderr, "Error ... Invalid value for option'-n'.\n");
tests/performance/nightly-tests/benchmarks/source/qsort_ new file +5
-0+ exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_array_len < MIN_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe a "
tests/performance/nightly-tests/benchmarks/source/qsort_ new file +6
-0+ "number less than %d.\n", MIN_ARRAY_LEN); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_array_len > MAX_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannotbe "
tests/performance/nightly-tests/benchmarks/source/qsort_ new file +26
-0+ "more than %d.\n", MAX_ARRAY_LEN); + exit(EXIT_FAILURE); + } + array_len = user_array_len; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for the array */ + array_to_be_sorted = (double *) malloc(array_len * sizeof(double)); + + /* Populate the_array with random numbers */ + srand(1); + for (size_t i = 0; i < array_len; i++) { + array_to_be_sorted[i] = range_factor * (double)rand(); + } + + /* Sort the_array using qsort() */ + qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]), + compare); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf("%14.10f %14.10f %14.10f\n", + array_to_be_sorted[0], array_to_be_sorted[1],array_to_be_sorted[2]);
tests/performance/nightly-tests/benchmarks/source/qsort_ new file +4
-0+ + /* Free all previously allocated space */ + free(array_to_be_sorted); +}tests/performance/nightly-tests/benchmarks/source/qsort_ renamed from tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c +0
-0
int32/qsort_int32.c
new file mode 100644
index 0000000000..76ca9c3490tests/performance/nightly-tests/benchmarks/source/qsort_ new file +0
-0
int32/qsort_int32.ctests/performance/nightly-tests/benchmarks/source/qsort_ new file +9
-0@@ -0,0 +1,103 @@ +/* + * Source file of a benchmark program involving sorting of an array + * of length n whose elements are "int32_t". The default value for n + * is 300000, and it can be set via command line as well. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman <[email protected]> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *+ * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details.+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */tests/performance/nightly-tests/benchmarks/source/qsort_ new file +38
-0+ +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> + +/* Number of elements in the array to be sorted */ +#define DEFAULT_ARRAY_LEN 300000 +#define MIN_ARRAY_LEN 3 +#define MAX_ARRAY_LEN 30000000 + +/* Upper limit used for generation of random numbers */ +#define UPPER_LIMIT 50000000 + +/* Comparison function passed to qsort() */ +static int compare(const void *a, const void *b) +{ + if (*(const int32_t *)a > *(const int32_t *)b) { + return 1; + } else if (*(const int32_t *)a < *(const int32_t *)b) { + return -1; + } + return 0; +} + +void main(int argc, char *argv[]) +{ + int32_t *array_to_be_sorted; + int32_t array_len = DEFAULT_ARRAY_LEN; + int32_t option; + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_array_len = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_array_len == 0) { + fprintf(stderr, "Error ... Invalid value for option'-n'.\n");
tests/performance/nightly-tests/benchmarks/source/qsort_ new file +5
-0+ exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_array_len < MIN_ARRAY_LEN) { +
Re: [PATCH 7/9] tests/performance: Add nightly tests
In reply to Ahmed KaramanQuoted 88 lines · Ahmed Karaman
A nightly performance testing system to monitor any change in QEMU performance across seventeen different targets.
The system includes eight different benchmarks to provide a variety of testing workloads.
dijkstra_double:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are double values. The value of n can be specified using the -n flag. The default value is 2000.
dijkstra_int32:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are int32 values. The value of n can be specified using the -n flag. The default value is 2000.
matmult_double:
Standard matrix multiplication of an n*n matrix of randomly generated double numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
matmult_int32:
Standard matrix multiplication of an n*n matrix of randomly generated integer numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
qsort_double:
Quick sort of an array of n randomly generated double numbers from 0 to 1000. The value of n is passed as an argument with the -n flag. The default value is 300000.
qsort_int32:
Quick sort of an array of n randomly generated integer numbers from 0 to 50000000. The value of n is passed as an argument with the -n flag.The default value is 300000.
qsort_string:
Quick sort of an array of 10000 randomly generated strings of size 8 (including null terminating character). The sort process is repeated n number of times. The value of n is passed as an argument with the -n flag. The default value is 20.
search_string:
Search for the occurrence of a small string in a much larger random string (“needle in a hay”). The search process is repeated n number of times and each time, a different large random string (“hay”) is generated. The value of n can be specified using the -n flag. The default value is 20.Syntax:
nightly_tests_core.py[-h] [-r REF]
Optional arguments:
-h, --help Show this help message and exit
-r REF, --reference REF
Reference QEMU version - Default is v5.1.0
Example of usage:
nightly_tests_core.py-r v5.1.0 2>log.txtThe following report includes detailed setup and execution details of the system:
https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/Signed-off-by Ahmed Karaman <[email protected]>--- tests/performance/nightly-tests/README.md | 243 +++++ .../source/dijkstra_double/dijkstra_double.c | 194 ++++ .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ .../source/matmult_double/matmult_double.c | 123 +++ .../source/matmult_int32/matmult_int32.c | 121 +++ .../source/qsort_double/qsort_double.c | 104 ++ .../source/qsort_int32/qsort_int32.c | 103 ++ .../source/qsort_string/qsort_string.c | 122 +++ .../source/search_string/search_string.c | 110 +++ .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ .../scripts/run_nightly_tests.py | 135 +++ .../nightly-tests/scripts/send_email.py | 56 ++ 12 files changed, 2423 insertions(+) create mode 100644 tests/performance/nightly-tests/README.md create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c create mode 100644
Quoted 1 line
tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
Perhaps we could compress these paths down to:
tests/tcg/benchmarks/foo.c
tests/tcg/benchmarks/bar.c
and then we can also ensure they are built using the existing TCG tests cross compile framework.
Quoted 17 lines
create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py create mode 100644 tests/performance/nightly-tests/scripts/send_email.pytests/performance/nightly-tests/README.md new file +7
-0@@ -0,0 +1,243 @@ +### QEMU Nightly Tests + +**Required settings:** + +Update the `GMAIL_USER` object in `send_email.py` with your credentials. + +For more details on how the system works, please check the [eighth
Quoted 2 lines
report](https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/) of the "TCG Continuos Benchmarking" series.
As external URLs are potentially unstable I think we want to distil the details into a rst do in docs/devel/
<snip>
Signature · 1 line
Alex Bennée
Re: [PATCH 7/9] tests/performance: Add nightly tests
Thanks Mr. Alex,
Quoted 95 lines · Alex Bennée
Quoted 88 lines · Ahmed Karaman
A nightly performance testing system to monitor any change in QEMU performance across seventeen different targets.
The system includes eight different benchmarks to provide a variety of testing workloads.
dijkstra_double:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are double values. The value of n can be specified using the -n flag. The default value is 2000.
dijkstra_int32:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are int32 values. The value of n can be specified using the -n flag. The default value is 2000.
matmult_double:
Standard matrix multiplication of an n*n matrix of randomly generated double numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
matmult_int32:
Standard matrix multiplication of an n*n matrix of randomly generated integer numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
qsort_double:
Quick sort of an array of n randomly generated double numbers from 0 to 1000. The value of n is passed as an argument with the -n flag. The default value is 300000.
qsort_int32:
Quick sort of an array of n randomly generated integer numbers from 0 to 50000000. The value of n is passed as an argument with the -n flag.The default value is 300000.
qsort_string:
Quick sort of an array of 10000 randomly generated strings of size 8 (including null terminating character). The sort process is repeated n number of times. The value of n is passed as an argument with the -n flag. The default value is 20.
search_string:
Search for the occurrence of a small string in a much larger random string (“needle in a hay”). The search process is repeated n number of times and each time, a different large random string (“hay”) is generated. The value of n can be specified using the -n flag. The default value is 20.Syntax:
nightly_tests_core.py[-h] [-r REF]
Optional arguments:
-h, --help Show this help message and exit
-r REF, --reference REF
Reference QEMU version - Default is v5.1.0
Example of usage:
nightly_tests_core.py-r v5.1.0 2>log.txtThe following report includes detailed setup and execution details of the system:
https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/Signed-off-by Ahmed Karaman <[email protected]>--- tests/performance/nightly-tests/README.md | 243 +++++ .../source/dijkstra_double/dijkstra_double.c | 194 ++++ .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ .../source/matmult_double/matmult_double.c | 123 +++ .../source/matmult_int32/matmult_int32.c | 121 +++ .../source/qsort_double/qsort_double.c | 104 ++ .../source/qsort_int32/qsort_int32.c | 103 ++ .../source/qsort_string/qsort_string.c | 122 +++ .../source/search_string/search_string.c | 110 +++ .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ .../scripts/run_nightly_tests.py | 135 +++ .../nightly-tests/scripts/send_email.py | 56 ++ 12 files changed, 2423 insertions(+) create mode 100644 tests/performance/nightly-tests/README.md create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c create mode 100644Quoted 1 line
tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
Quoted 8 lines
Perhaps we could compress these paths down to:
tests/tcg/benchmarks/foo.c
tests/tcg/benchmarks/bar.cand then we can also ensure they are built using the existing TCG tests cross compile framework.
Can you provide the commands for compiling the benchmarks for the 17 targets?
Quoted 14 lines
Quoted 14 lines
create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py create mode 100644 tests/performance/nightly-tests/scripts/send_email.pytests/performance/nightly-tests/README.md new file +4
-0@@ -0,0 +1,243 @@ +### QEMU Nightly Tests + +**Required settings:** +
Quoted 8 lines
Quoted 5 lines
tests/performance/nightly-tests/README.md new file +3
-0+Update the `GMAIL_USER` object in `send_email.py` with your credentials. + +For more details on how the system works, please check the [eighthreport](https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/) of the "TCG Continuos Benchmarking" series.
As external URLs are potentially unstable I think we want to distil the details into a rst do in docs/devel/
Any advice on converting markdown to rst? Are there any specific rules that should be followed to write rst documentation for QEMU.
Quoted 5 lines
<snip>
--
Alex Bennée
Best regards,
Ahmed Karaman
Re: [PATCH 7/9] tests/performance: Add nightly tests
In reply to Alex BennéeQuoted 102 lines · Alex Bennée
Quoted 60 lines · Ahmed Karaman
A nightly performance testing system to monitor any change in QEMU performance across seventeen different targets.
The system includes eight different benchmarks to provide a variety of testing workloads.
dijkstra_double:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are double values. The value of n can be specified using the -n flag. The default value is 2000.
dijkstra_int32:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are int32 values. The value of n can be specified using the -n flag. The default value is 2000.
matmult_double:
Standard matrix multiplication of an n*n matrix of randomly generated double numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
matmult_int32:
Standard matrix multiplication of an n*n matrix of randomly generated integer numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
qsort_double:
Quick sort of an array of n randomly generated double numbers from 0 to 1000. The value of n is passed as an argument with the -n flag. The default value is 300000.
qsort_int32:
Quick sort of an array of n randomly generated integer numbers from 0 to 50000000. The value of n is passed as an argument with the -n flag.The default value is 300000.
qsort_string:
Quick sort of an array of 10000 randomly generated strings of size 8 (including null terminating character). The sort process is repeated n number of times. The value of n is passed as an argument with the -n flag. The default value is 20.
search_string:
Search for the occurrence of a small string in a much larger random string (“needle in a hay”). The search process is repeated n number of times and each time, a different large random string (“hay”) is generated. The value of n can be specified using the -n flag. The default value is 20.Syntax:
nightly_tests_core.py[-h] [-r REF]
Optional arguments:
-h, --help Show this help message and exit
-r REF, --reference REF
Reference QEMU version - Default is v5.1.0
Example of usage:
nightly_tests_core.py-r v5.1.0 2>log.txtQuoted 3 lines
The following report includes detailed setup and execution details of the system:
https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/
Quoted 18 lines
Signed-off-by Ahmed Karaman <[email protected]>--- tests/performance/nightly-tests/README.md | 243 +++++ .../source/dijkstra_double/dijkstra_double.c | 194 ++++ .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ .../source/matmult_double/matmult_double.c | 123 +++ .../source/matmult_int32/matmult_int32.c | 121 +++ .../source/qsort_double/qsort_double.c | 104 ++ .../source/qsort_int32/qsort_int32.c | 103 ++ .../source/qsort_string/qsort_string.c | 122 +++ .../source/search_string/search_string.c | 110 +++ .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ .../scripts/run_nightly_tests.py | 135 +++ .../nightly-tests/scripts/send_email.py | 56 ++ 12 files changed, 2423 insertions(+) create mode 100644 tests/performance/nightly-tests/README.md create mode 100644 tests/performance/nightly-tests/benchmarks/source/
dijkstra_double/dijkstra_double.cQuoted 1 line
create mode 100644 tests/performance/nightly-tests/benchmarks/source/
dijkstra_int32/dijkstra_int32.cQuoted 1 line
create mode 100644 tests/performance/nightly-tests/benchmarks/source/
matmult_double/matmult_double.cQuoted 1 line
create mode 100644 tests/performance/nightly-tests/benchmarks/source/
matmult_int32/matmult_int32.cQuoted 1 line
create mode 100644 tests/performance/nightly-
tests/benchmarks/source/qsort_double/qsort_double.cQuoted 1 line
create mode 100644 tests/performance/nightly-
tests/benchmarks/source/qsort_int32/qsort_int32.cQuoted 1 line
create mode 100644 tests/performance/nightly-
tests/benchmarks/source/qsort_string/qsort_string.cQuoted 2 lines
create mode 100644tests/performance/nightly-tests/benchmarks/source/
search_string/search_string.c
Quoted 9 lines
Perhaps we could compress these paths down to:
tests/tcg/benchmarks/foo.c
tests/tcg/benchmarks/bar.cand then we can also ensure they are built using the existing TCG tests cross compile framework.
Hi,
May I just bring some alternative views on this topic of benchmarks/multiple cross-compiling/nightly tests?
Having a known source code of any benchmark (and also license-compatible with QEMU) is, for sure, a good thing. However, for actual test execution, source code is not relevant, but the executables built for a diverse set of targets. Expectation that a test bed should be able to build let's say 17 version for each benchmark is a tall order - and, in fact, unnecessary self-imposed limitation.
My suggestion is that 17x8 executables needed for nightly tests in its current form should be prebuilt - and kept permanently in a separate repository (just not to burden QEMU repository). They are anyway built only once, there is no need to update them at all.
This will allow nightly test execution on any system that can execute qemu, without any prerequisite for cross-compiling.
Their source code could and should be integrated into QEMU tree, at the place of choice, but the test beds should not be subject to any burden of cross-compiling, simply because there is no need for that.
Yours,
Aleksandar
Quoted 33 lines
Quoted 1 line
create mode 100755 tests/performance/nightly-
tests/scripts/nightly_tests_core.pyQuoted 1 line
create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_
tests.pyQuoted 1 line
create mode 100644 tests/performance/nightly-
tests/scripts/send_email.pyReview
Quoted 2 lines
(quoted diff) +0
-0
b/tests/performance/nightly-tests/README.mdQuoted 12 lines
new file mode 100644
index 0000000000..6db3b351b3tests/performance/nightly-tests/README.md new file +7
-0@@ -0,0 +1,243 @@ +### QEMU Nightly Tests + +**Required settings:** + +Update the `GMAIL_USER` object in `send_email.py` with your credentials. + +For more details on how the system works, please check the [eighthQuoted 1 line
Benchmarking/QEMU-Nightly-Performance-Tests/)
Quoted 1 line
of the "TCG Continuos Benchmarking" series.
As external URLs are potentially unstable I think we want to distil the details into a rst do in docs/devel/
<snip>
--
Alex Bennée
Re: [PATCH 7/9] tests/performance: Add nightly tests
Quoted 130 lines · Aleksandar Markovic
Quoted 102 lines · Alex Bennée
Quoted 60 lines · Ahmed Karaman
A nightly performance testing system to monitor any change in QEMU performance across seventeen different targets.
The system includes eight different benchmarks to provide a variety of testing workloads.
dijkstra_double:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are double values. The value of n can be specified using the -n flag. The default value is 2000.
dijkstra_int32:
Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are int32 values. The value of n can be specified using the -n flag. The default value is 2000.
matmult_double:
Standard matrix multiplication of an n*n matrix of randomly generated double numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
matmult_int32:
Standard matrix multiplication of an n*n matrix of randomly generated integer numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200.
qsort_double:
Quick sort of an array of n randomly generated double numbers from 0 to 1000. The value of n is passed as an argument with the -n flag. The default value is 300000.
qsort_int32:
Quick sort of an array of n randomly generated integer numbers from 0 to 50000000. The value of n is passed as an argument with the -n flag.The default value is 300000.
qsort_string:
Quick sort of an array of 10000 randomly generated strings of size 8 (including null terminating character). The sort process is repeated n number of times. The value of n is passed as an argument with the -n flag. The default value is 20.
search_string:
Search for the occurrence of a small string in a much larger random string (“needle in a hay”). The search process is repeated n number of times and each time, a different large random string (“hay”) is generated. The value of n can be specified using the -n flag. The default value is 20.Syntax:
nightly_tests_core.py[-h] [-r REF]
Optional arguments:
-h, --help Show this help message and exit
-r REF, --reference REF
Reference QEMU version - Default is v5.1.0
Example of usage:
nightly_tests_core.py-r v5.1.0 2>log.txtQuoted 3 lines
The following report includes detailed setup and execution details of the system:
https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/
Quoted 18 lines
Signed-off-by Ahmed Karaman <[email protected]>--- tests/performance/nightly-tests/README.md | 243 +++++ .../source/dijkstra_double/dijkstra_double.c | 194 ++++ .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ .../source/matmult_double/matmult_double.c | 123 +++ .../source/matmult_int32/matmult_int32.c | 121 +++ .../source/qsort_double/qsort_double.c | 104 ++ .../source/qsort_int32/qsort_int32.c | 103 ++ .../source/qsort_string/qsort_string.c | 122 +++ .../source/search_string/search_string.c | 110 +++ .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ .../scripts/run_nightly_tests.py | 135 +++ .../nightly-tests/scripts/send_email.py | 56 ++ 12 files changed, 2423 insertions(+) create mode 100644 tests/performance/nightly-tests/README.md create mode 100644 tests/performance/nightly-tests/benchmarks/source/
dijkstra_double/dijkstra_double.cQuoted 1 line
create mode 100644 tests/performance/nightly-tests/benchmarks/source/
dijkstra_int32/dijkstra_int32.cQuoted 1 line
create mode 100644 tests/performance/nightly-tests/benchmarks/source/
matmult_double/matmult_double.cQuoted 1 line
create mode 100644 tests/performance/nightly-tests/benchmarks/source/
matmult_int32/matmult_int32.cQuoted 1 line
create mode 100644 tests/performance/nightly-
tests/benchmarks/source/qsort_double/qsort_double.cQuoted 1 line
create mode 100644 tests/performance/nightly-
tests/benchmarks/source/qsort_int32/qsort_int32.cQuoted 1 line
create mode 100644 tests/performance/nightly-
tests/benchmarks/source/qsort_string/qsort_string.cQuoted 2 lines
create mode 100644tests/performance/nightly-tests/benchmarks/source/
search_string/search_string.cQuoted 9 lines
Perhaps we could compress these paths down to:
tests/tcg/benchmarks/foo.c
tests/tcg/benchmarks/bar.cand then we can also ensure they are built using the existing TCG tests cross compile framework.
Hi,
May I just bring some alternative views on this topic of benchmarks/multiple cross-compiling/nightly tests?
Having a known source code of any benchmark (and also license-compatible with QEMU) is, for sure, a good thing. However, for actual test execution, source code is not relevant, but the executables built for a diverse set of targets. Expectation that a test bed should be able to build let's say 17 version for each benchmark is a tall order - and, in fact, unnecessary self-imposed limitation.
My suggestion is that 17x8 executables needed for nightly tests in its current form should be prebuilt - and kept permanently in a separate repository (just not to burden QEMU repository). They are anyway built only once, there is no need to update them at all.
Quoted 6 lines
This will allow nightly test execution on any system that can execute qemu, without any prerequisite for cross-compiling.
Their source code could and should be integrated into QEMU tree, at the place of choice, but the test beds should not be subject to any burden of cross-compiling, simply because there is no need for that.
I have no problem with the test binaries being stored somewhere and accessed when needed - as long as we have a source build-able version in the tree. It would seem silly not to at least integrate the build for the 19 different cross compilers we already support for the check-tcg tests.
I'm not proposing we run them every test either, but at least have the option to do:
make build-tcg-benchmark
make run-tcg-benchmark
so it's easy for a developer to get to a state when they can tinker and debug.
Quoted 40 lines
Yours,
AleksandarQuoted 33 lines
Quoted 1 line
create mode 100755 tests/performance/nightly-
tests/scripts/nightly_tests_core.pyQuoted 1 line
create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_
tests.pyQuoted 1 line
create mode 100644 tests/performance/nightly-
tests/scripts/send_email.pyReview
Quoted 2 lines
(quoted diff) +0
-0
b/tests/performance/nightly-tests/README.mdQuoted 12 lines
new file mode 100644
index 0000000000..6db3b351b3tests/performance/nightly-tests/README.md new file +7
-0@@ -0,0 +1,243 @@ +### QEMU Nightly Tests + +**Required settings:** + +Update the `GMAIL_USER` object in `send_email.py` with your credentials. + +For more details on how the system works, please check the [eighthQuoted 1 line
Benchmarking/QEMU-Nightly-Performance-Tests/)
Quoted 1 line
of the "TCG Continuos Benchmarking" series.
As external URLs are potentially unstable I think we want to distil the details into a rst do in docs/devel/
<snip>
--
Alex Bennée
Signature · 1 line
Alex Bennée
PATCH 8/9 MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests' subsection
In reply to Ahmed KaramanAdd a new 'tests/performance' directory under the 'Performance Tools and Tests' subsection.
--- MAINTAINERS | 1 + 1 file changed, 1 insertion(+)
MAINTAINERS +1 -0
@@ -3124,3 +3124,4 @@ Performance Tools and Tests M: Ahmed Karaman <[email protected]> S: Maintained F: scripts/performance/ +F: tests/performance/
Signature · 1 line
2.17.1
Re: [PATCH 8/9] MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests' subsection
Quoted 4 lines · Ahmed Karaman
Add a new 'tests/performance' directory under the 'Performance Tools and Tests' subsection.
Quoted 18 lines
Signed-off-by Ahmed Karaman <[email protected]>--- MAINTAINERS | 1 + 1 file changed, 1 insertion(+)MAINTAINERS +1
-1@@ -3124,3 +3124,4 @@ Performance Tools and Tests M: Ahmed Karaman <[email protected]> S: Maintained F: scripts/performance/ +F: tests/performance/ --2.17.1
PATCH 9/9 scripts/performance: Add topN_system.py script
In reply to Ahmed KaramanPython script for listing the topN executed QEMU functions in system mode.
Syntax:topN_system.py [-h] [-n <number of displayed top functions>] -- \
<qemu system executable> <qemu system options>
[-h] - Print the script arguments help message.
[-n] - Specify the number of top functions to print.
- If this flag is not specified, the tool defaults to 25.
Example of usage:topN_system.py -n 20 -- qemu-system-x86_64 -m 1024 -kernel <kernel> \
-initrd <initrd>
Example output:
Number of instructions: 150,991,381,071
No. Percentage Name --- ---------- ------------------------------ 1 11.30% helper_lookup_tb_ptr 2 7.01% liveness_pass_1 3 4.48% tcg_gen_code 4 3.41% tcg_optimize 5 1.84% tcg_out_opc.isra.13 6 1.78% helper_pcmpeqb_xmm 7 1.20% object_dynamic_cast_assert 8 1.00% cpu_exec 9 0.99% tcg_temp_new_internal 10 0.88% tb_htable_lookup 11 0.84% object_class_dynamic_cast_assert 12 0.81% init_ts_info 13 0.80% tlb_set_page_with_attrs 14 0.77% victim_tlb_hit 15 0.75% tcg_out_sib_offset 16 0.62% tcg_op_alloc 17 0.61% helper_pmovmskb_xmm 18 0.58% disas_insn.isra.50 19 0.56% helper_pcmpgtb_xmm 20 0.56% address_space_ldq
--- scripts/performance/topN_system.py | 158 +++++++++++++++++++++++++++++ 1 file changed, 158 insertions(+) create mode 100755 scripts/performance/topN_system.py
scripts/performance/topN_system.py new file +158 -0
@@ -0,0 +1,158 @@ +#!/usr/bin/env python3 + +""" +Print the top N most executed functions in QEMU system mode emulation. + +Syntax: +topN_system.py [-h] [-n <number of displayed top functions>] -- \ + <qemu system executable> <qemu system options> + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_system.py -n 20 -- qemu-system-x86_64 -m 1024 -kernel <kernel> \ + -initrd <initrd> + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman <[email protected]> +Copyright (C) 2020 Aleksandar Markovic <[email protected]> + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <https://www.gnu.org/licenses/>. +""" + +import argparse +import os +import subprocess +import sys +import tempfile + + +# Parse the command line arguments +PARSER = argparse.ArgumentParser( + usage="usage: topN_system.py [-h] [-n <number of displayed top functions>]" + " -- <qemu system executable> <qemu system options>") + +PARSER.add_argument("-n", dest="top", type=int, default=25, + help="Specify the number of top functions to print.") + +PARSER.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS) + +ARGS = PARSER.parse_args() + +# Extract the needed variables from the args +COMMAND = ARGS.command +TOP = ARGS.top + +# Ensure that perf is installed +CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"], + stdout=subprocess.DEVNULL, + check=False) +if CHECK_PERF_PRESENCE.returncode: + sys.exit("Please install perf before running the script!") + +# Ensure user has previllage to run perf +CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + check=False) +if CHECK_PERF_EXECUTABILITY.returncode: + sys.exit(""" +Error: +You may not have permission to collect stats. +Consider tweaking /proc/sys/kernel/perf_event_paranoid, +which controls use of the performance events system by +unprivileged users (without CAP_SYS_ADMIN). + -1: Allow use of (almost) all events by all users + Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK + 0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN + Disallow raw tracepoint access by users without CAP_SYS_ADMIN + 1: Disallow CPU event access by users without CAP_SYS_ADMIN + 2: Disallow kernel profiling by users without CAP_SYS_ADMIN +To make this setting permanent, edit /etc/sysctl.conf too, e.g.: + kernel.perf_event_paranoid = -1 + +* Alternatively, you can run this script under sudo privileges. +""") + +# Run perf and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + RECORD_PATH = os.path.join(tmpdir, "record.data") + REPORT_PATH = os.path.join(tmpdir, "report.txt") + + PERF_RECORD = subprocess.run((["perf", + "record", + "-e", + "instructions", + "--output="+RECORD_PATH] + + COMMAND), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if PERF_RECORD.returncode: + sys.exit(PERF_RECORD.stderr.decode("utf-8")) + + with open(REPORT_PATH, "w") as output: + PERF_REPORT = subprocess.run(["perf", + "report", + "--input=" + RECORD_PATH, + "--stdio"], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if PERF_REPORT.returncode: + sys.exit(PERF_REPORT.stderr.decode("utf-8")) + + # Save the reported data to FUNCTIONS[] and INSTRUCTIONS + with open(REPORT_PATH, "r") as data: + LINES = data.readlines() + # Read the number of instructions + INSTRUCTIONS = int(LINES[6].split()[-1]) + # Continue reading: + # Only read lines that are not empty + # Only read lines that are not comments (comments start with #) + # Only read functions executed by qemu-system + FUNCTIONS = [line for line in LINES if line + and line[0] != "\n" + and line[0] != "#" + and "qemu-system" in line.split()[2]] + + +# Limit the number of top functions to "TOP" +NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS) + +# Store the data of the top functions in TOP_FUNCTIONS[] +TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS] + +# Print total instructions +print("\nNumber of instructions:", format(INSTRUCTIONS, ",")) +# Print table header +print("\n{:>4} {:>10} {}\n{} {} {}".format("No.", + "Percentage", + "Name", + "-" * 4, + "-" * 10, + "-" * 30)) + +# Print top N functions +for (index, function) in enumerate(TOP_FUNCTIONS, start=1): + function_data = function.split() + function_percentage = function_data[0] + function_name = function_data[-1] + function_invoker = " ".join(function_data[2:-2]) + print("{:>4} {:>10} {}".format(index, + function_percentage, + function_name))
Signature · 1 line
2.17.1