workflows: add fio-tests performance tests
PATCH v2 1/1 workflows: add fio-tests performance tests
I had written fio-tests [0] a long time ago but although it used Kconfig, at that point I had lacked the for-sight of leveraging jinja2 / ansible to make things more declarative. It's been on my back log to try to get that ported over to kdevops but now with Claude Code, it just required a series of prompts. And its even better now. *This* is how we scale.
I've added a demo tree which just has the graphs for those just itching to see what this produces [1]. It's just a demo comparing two separate runs don't get too excited as its just a silly guest with virtio drives. However this should hopefully show you how easily and quickly with the new A/B testing feature.
The run time for tests is configurable, you also use the FIO_QUICK environment variable to do quick tests.
The runtime choices are:
- Default: 60 seconds runtime, 10 seconds ramp time
- Quick: 10 seconds runtime, 2 seconds ramp time (selected with FIO_QUICK=y)
- Custom High: 300 seconds runtime, 30 seconds ramp time
- Custom Low: 5 seconds runtime, 1 second ramp time
We add defconfig-fio-tests-perf which to enable all performance testing knobs:
- All block sizes (4K, 8K, 16K, 32K, 64K, 128K)
- All IO depths (1, 4, 8, 16, 32, 64)
- All job counts (1, 2, 4, 8, 16)
- All test patterns (random/sequential read/write, mixed workloads)
- High DPI (300) for graphs
- A/B testing with baseline and dev nodes
This allows quick CI testing with:
make defconfig-fio-tests-perf FIO_QUICK=y
Or comprehensive performance testing with:
make defconfig-fio-tests-perf
Key features:
- Configurable test matrix: block sizes (4K-128K), IO depths (1-64), job counts
- Multiple workload patterns: random/sequential read/write, mixed workloads
- Advanced configuration: IO engines, direct IO, fsync options
- Performance logging: bandwidth, IOPS, and latency metrics
- Baseline management and results analysis
- FIO_TESTS_ENABLE_GRAPHING: Enable/disable graphing capabilities
- Graph format, DPI, and theme configuration options
- Updated CI defconfig with graphing support (150 DPI for faster CI)
Key graphing features support:
- Performance analysis: bandwidth heatmaps, IOPS scaling, latency distributions
- A/B comparison: baseline vs development configuration analysis
- Trend analysis: block size scaling, IO depth optimization, correlation matrices
- Configurable output: PNG/SVG/PDF formats, DPI settings, matplotlib themes
Documentation:
- docs/fio-tests.md: Comprehensive workflow documentation covering:
* Origin story and relationship to upstream fio-tests project
* Quick start and configuration examples
* Test matrix configuration and device setup
* A/B testing and baseline management
* Graphing and visualization capabilities
* CI integration and troubleshooting guides
* Best practices and performance considerations
A minimal CI configuration (defconfig-fio-tests-ci) enables automated testing in GitHub Actions using /dev/null as the target device with a reduced test matrix for fast execution.
We extend PROMPTS.md with prompts used for all this.
Usage:
make defconfig-fio-tests-ci # Simple CI testing make menuconfig # Interactive configuration make fio-tests # Run performance tests make fio-tests-baseline # Establish baseline make fio-tests-results # Collect results make fio-tests-graph # Generate performance graphs make fio-tests-compare # Compare baseline vs dev results make fio-tests-trend-analysis # Analyze performance trends
---
Changes on this v2:
- Rebased
- The last graphs didn't make too much sense so I've asked for some new ones. I think we can start with this for now.
- Removed the docker tests as we can't run make menuconfig on a docker container. We can later enable on bare metal testing with the defconfigs/fio-tests-ci. It should be fast.
.gitignore | 2 + PROMPTS.md | 105 ++++ README.md | 10 + defconfigs/fio-tests-ci | 56 ++ defconfigs/fio-tests-perf | 58 +++ docs/fio-tests.md | 377 ++++++++++++++ kconfigs/workflows/Kconfig | 27 + playbooks/fio-tests-baseline.yml | 29 ++ playbooks/fio-tests-compare.yml | 33 ++ playbooks/fio-tests-graph.yml | 78 +++ playbooks/fio-tests-results.yml | 10 + playbooks/fio-tests-trend-analysis.yml | 30 ++ playbooks/fio-tests.yml | 11 + .../python/workflows/fio-tests/fio-compare.py | 383 ++++++++++++++ .../python/workflows/fio-tests/fio-plot.py | 350 +++++++++++++ .../workflows/fio-tests/fio-trend-analysis.py | 477 ++++++++++++++++++ playbooks/roles/fio-tests/defaults/main.yml | 48 ++ .../tasks/install-deps/debian/main.yml | 20 + .../fio-tests/tasks/install-deps/main.yml | 3 + .../tasks/install-deps/redhat/main.yml | 20 + .../tasks/install-deps/suse/main.yml | 20 + playbooks/roles/fio-tests/tasks/main.yaml | 170 +++++++ .../roles/fio-tests/templates/fio-job.ini.j2 | 29 ++ playbooks/roles/gen_hosts/tasks/main.yml | 13 + .../roles/gen_hosts/templates/fio-tests.j2 | 28 + playbooks/roles/gen_hosts/templates/hosts.j2 | 38 ++ playbooks/roles/gen_nodes/tasks/main.yml | 32 ++ workflows/Makefile | 4 + workflows/fio-tests/Kconfig | 420 +++++++++++++++ workflows/fio-tests/Makefile | 68 +++ 30 files changed, 2949 insertions(+) create mode 100644 defconfigs/fio-tests-ci create mode 100644 defconfigs/fio-tests-perf create mode 100644 docs/fio-tests.md create mode 100644 playbooks/fio-tests-baseline.yml create mode 100644 playbooks/fio-tests-compare.yml create mode 100644 playbooks/fio-tests-graph.yml create mode 100644 playbooks/fio-tests-results.yml create mode 100644 playbooks/fio-tests-trend-analysis.yml create mode 100644 playbooks/fio-tests.yml create mode 100755 playbooks/python/workflows/fio-tests/fio-compare.py create mode 100755 playbooks/python/workflows/fio-tests/fio-plot.py create mode 100755 playbooks/python/workflows/fio-tests/fio-trend-analysis.py create mode 100644 playbooks/roles/fio-tests/defaults/main.yml create mode 100644 playbooks/roles/fio-tests/tasks/install-deps/debian/main.yml create mode 100644 playbooks/roles/fio-tests/tasks/install-deps/main.yml create mode 100644 playbooks/roles/fio-tests/tasks/install-deps/redhat/main.yml create mode 100644 playbooks/roles/fio-tests/tasks/install-deps/suse/main.yml create mode 100644 playbooks/roles/fio-tests/tasks/main.yaml create mode 100644 playbooks/roles/fio-tests/templates/fio-job.ini.j2 create mode 100644 playbooks/roles/gen_hosts/templates/fio-tests.j2 create mode 100644 workflows/fio-tests/Kconfig create mode 100644 workflows/fio-tests/Makefile
.gitignore +2 -0
@@ -70,6 +70,8 @@ workflows/sysbench/results/ workflows/mmtests/results/ tmp +workflows/fio-tests/results/ + playbooks/roles/linux-mirror/linux-mirror-systemd/*.service playbooks/roles/linux-mirror/linux-mirror-systemd/*.timer playbooks/roles/linux-mirror/linux-mirror-systemd/mirrors.yaml
PROMPTS.md +105 -0
111 lines not on this page · raw message
README.md +10 -0
@@ -13,6 +13,7 @@ Table of Contents * [CXL](#cxl) * [reboot-limit](#reboot-limit) * [sysbench](#sysbench) + * [fio-tests](#fio-tests) * [kdevops chats](#kdevops-chats) * [kdevops on discord](#kdevops-on-discord) * [kdevops IRC](#kdevops-irc)
@@ -263,6 +264,15 @@ kdevops supports automation of sysbench tests on VMs with or without providers. For details refer to the [kdevops sysbench documentation](docs/sysbench/sysbench.md). +### fio-tests + +kdevops includes comprehensive storage performance testing through the fio-tests +workflow, adapted from the original [fio-tests framework](https://github.com/mcgrof/fio-tests). +This workflow provides flexible I/O benchmarking with configurable test matrices, +A/B testing capabilities, and advanced graphing and visualization support. For +detailed configuration and usage information, refer to the +[kdevops fio-tests documentation](docs/fio-tests.md). + ## kdevops chats We use discord and IRC. Right now we have more folks on discord than on IRC.
defconfigs/fio-tests-ci new file +56 -0
56 lines not on this page · raw message
defconfigs/fio-tests-perf new file +58 -0
58 lines not on this page · raw message
docs/fio-tests.md new file +377 -0
377 lines not on this page · raw message
kconfigs/workflows/Kconfig +27 -0
@@ -207,6 +207,13 @@ config KDEVOPS_WORKFLOW_DEDICATE_MMTESTS This will dedicate your configuration to running only the mmtests workflow for memory fragmentation testing. +config KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS + bool "fio-tests" + select KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS + help + This will dedicate your configuration to running only the + fio-tests workflow for comprehensive storage performance testing. + endchoice config KDEVOPS_WORKFLOW_NAME
@@ -221,6 +228,7 @@ config KDEVOPS_WORKFLOW_NAME default "nfstest" if KDEVOPS_WORKFLOW_DEDICATE_NFSTEST default "sysbench" if KDEVOPS_WORKFLOW_DEDICATE_SYSBENCH default "mmtests" if KDEVOPS_WORKFLOW_DEDICATE_MMTESTS + default "fio-tests" if KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS endif
@@ -322,6 +330,14 @@ config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_MMTESTS Select this option if you want to provision mmtests on a single target node for by-hand testing. +config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_FIO_TESTS + bool "fio-tests" + select KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS + depends on LIBVIRT || TERRAFORM_PRIVATE_NET + help + Select this option if you want to provision fio-tests on a + single target node for by-hand testing. + endif # !WORKFLOWS_DEDICATED_WORKFLOW config KDEVOPS_WORKFLOW_ENABLE_FSTESTS
@@ -435,6 +451,17 @@ source "workflows/mmtests/Kconfig" endmenu endif # KDEVOPS_WORKFLOW_ENABLE_MMTESTS +config KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS + bool + output yaml + default y if KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_FIO_TESTS || KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS + +if KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS +menu "Configure and run fio-tests" +source "workflows/fio-tests/Kconfig" +endmenu +endif # KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS + config KDEVOPS_WORKFLOW_ENABLE_SSD_STEADY_STATE bool "Attain SSD steady state prior to tests" output yaml
playbooks/fio-tests-baseline.yml new file +29 -0
@@ -0,0 +1,29 @@ +--- +- hosts: + - baseline + - dev + become: no + vars: + ansible_ssh_pipelining: True + tasks: + - name: Create baseline directory structure + file: + path: "{{ fio_tests_results_dir }}/baseline" + state: directory + mode: '0755' + become: yes + + - name: Save current test configuration as baseline + copy: + src: "{{ fio_tests_results_dir }}/results_{{ item }}.txt" + dest: "{{ fio_tests_results_dir }}/baseline/baseline_{{ item }}.txt" + remote_src: yes + backup: yes + with_fileglob: + - "{{ fio_tests_results_dir }}/results_*.txt" + become: yes + ignore_errors: yes + + - name: Create baseline timestamp + shell: date > "{{ fio_tests_results_dir }}/baseline/baseline_timestamp.txt" + become: yes
playbooks/fio-tests-compare.yml new file +33 -0
@@ -0,0 +1,33 @@ +--- +- hosts: localhost + become: no + vars: + ansible_ssh_pipelining: True + tasks: + - name: Check if baseline and dev hosts exist in inventory + fail: + msg: "Both baseline and dev hosts must exist for comparison" + when: "'baseline' not in groups or 'dev' not in groups" + + - name: Create local graph results comparison directory + file: + path: "{{ topdir_path }}/workflows/fio-tests/results/graphs" + state: directory + mode: '0755' + + - name: Generate comparison graphs + shell: | + python3 {{ topdir_path }}/playbooks/python/workflows/fio-tests/fio-compare.py \ + {{ topdir_path }}/workflows/fio-tests/results/{{ groups['baseline'][0] }}/fio-tests-results-{{ groups['baseline'][0] }} \ + {{ topdir_path }}/workflows/fio-tests/results/{{ groups['dev'][0] }}/fio-tests-results-{{ groups['dev'][0] }} \ + --output-dir {{ topdir_path }}/workflows/fio-tests/results/graphs \ + --baseline-label "Baseline" \ + --dev-label "Development" + + - name: List comparison results + shell: ls -la {{ topdir_path }}/workflows/fio-tests/results/graphs + register: comparison_list + + - name: Display comparison results + debug: + msg: "{{ comparison_list.stdout_lines }}"
playbooks/fio-tests-graph.yml new file +78 -0
78 lines not on this page · raw message
playbooks/fio-tests-results.yml new file +10 -0
@@ -0,0 +1,10 @@ +--- +- hosts: + - baseline + - dev + become: no + vars: + ansible_ssh_pipelining: True + roles: + - role: fio-tests + tags: ['results']
playbooks/fio-tests-trend-analysis.yml new file +30 -0
@@ -0,0 +1,30 @@ +--- +- hosts: + - baseline + - dev + become: no + vars: + ansible_ssh_pipelining: True + tasks: + - include_role: + name: create_data_partition + tags: [ 'oscheck', 'data_partition' ] + + - name: Generate fio trend analysis + shell: | + cd {{ fio_tests_results_dir }} + mkdir -p analysis + python3 {{ kdevops_data }}/playbooks/python/workflows/fio-tests/fio-trend-analysis.py \ + . --output-dir analysis + args: + creates: "{{ fio_tests_results_dir }}/analysis/block_size_trends.png" + become: yes + + - name: List generated analysis files + shell: ls -la {{ fio_tests_results_dir }}/analysis/ + become: yes + register: analysis_list + + - name: Display generated analysis files + debug: + msg: "{{ analysis_list.stdout_lines }}"
playbooks/fio-tests.yml new file +11 -0
@@ -0,0 +1,11 @@ +--- +- hosts: + - baseline + - dev + become: no + vars: + ansible_ssh_pipelining: True + roles: + - role: create_data_partition + tags: ['data_partition'] + - role: fio-tests
playbooks/python/workflows/fio-tests/fio-compare.py new file +383 -0
383 lines not on this page · raw message
playbooks/python/workflows/fio-tests/fio-plot.py new file +350 -0
350 lines not on this page · raw message
playbooks/python/workflows/fio-tests/fio-trend-analysis.py new file +477 -0
477 lines not on this page · raw message
playbooks/roles/fio-tests/defaults/main.yml new file +48 -0
48 lines not on this page · raw message
playbooks/roles/fio-tests/tasks/install-deps/debian/main.yml new file +20 -0
@@ -0,0 +1,20 @@ +--- +- name: Install fio for Debian/Ubuntu + package: + name: + - fio + - python3 + state: present + become: yes + +- name: Install graphing dependencies for Debian/Ubuntu + package: + name: + - python3-pip + - python3-pandas + - python3-matplotlib + - python3-seaborn + - python3-numpy + state: present + become: yes + when: fio_tests_enable_graphing is defined and fio_tests_enable_graphing
playbooks/roles/fio-tests/tasks/install-deps/main.yml new file +3 -0
@@ -0,0 +1,3 @@ +--- +- name: Include distribution-specific installation tasks + include_tasks: "{{ ansible_os_family | lower }}/main.yml"
playbooks/roles/fio-tests/tasks/install-deps/redhat/main.yml new file +20 -0
@@ -0,0 +1,20 @@ +--- +- name: Install fio for RHEL/CentOS/Fedora + package: + name: + - fio + - python3 + state: present + become: yes + +- name: Install graphing dependencies for RHEL/CentOS/Fedora + package: + name: + - python3-pip + - python3-pandas + - python3-matplotlib + - python3-seaborn + - python3-numpy + state: present + become: yes + when: fio_tests_enable_graphing is defined and fio_tests_enable_graphing
playbooks/roles/fio-tests/tasks/install-deps/suse/main.yml new file +20 -0
@@ -0,0 +1,20 @@ +--- +- name: Install fio for SUSE + package: + name: + - fio + - python3 + state: present + become: yes + +- name: Install graphing dependencies for SUSE + package: + name: + - python3-pip + - python3-pandas + - python3-matplotlib + - python3-seaborn + - python3-numpy + state: present + become: yes + when: fio_tests_enable_graphing is defined and fio_tests_enable_graphing
playbooks/roles/fio-tests/tasks/main.yaml new file +170 -0
170 lines not on this page · raw message
playbooks/roles/fio-tests/templates/fio-job.ini.j2 new file +29 -0
@@ -0,0 +1,29 @@ +[global] +ioengine={{ fio_tests_ioengine }} +{% if fio_tests_device == '/dev/null' %} +direct=0 +{% else %} +direct={{ fio_tests_direct | int }} +{% endif %} +{% if fio_tests_device == '/dev/null' %} +fsync_on_close=0 +{% else %} +fsync_on_close={{ fio_tests_fsync_on_close | int }} +{% endif %} +group_reporting=1 +time_based=1 +runtime={{ fio_tests_runtime }} +ramp_time={{ fio_tests_ramp_time }} + +[{{ pattern.name }}_bs{{ block_size }}_iodepth{{ io_depth }}_jobs{{ num_jobs }}] +filename={{ fio_tests_device }} +{% if fio_tests_device == '/dev/null' %} +size=1G +{% endif %} +rw={{ pattern.rw }} +{% if pattern.rwmixread is defined and pattern.rw in ['randrw', 'rw'] %} +rwmixread={{ pattern.rwmixread }} +{% endif %} +bs={{ block_size }} +iodepth={{ io_depth }} +numjobs={{ num_jobs }}
playbooks/roles/gen_hosts/tasks/main.yml +13 -0
@@ -324,6 +324,19 @@ - kdevops_workflow_enable_sysbench - ansible_hosts_template.stat.exists +- name: Generate the Ansible hosts file for a dedicated fio-tests setup + tags: [ 'hosts' ] + template: + src: "{{ kdevops_hosts_template }}" + dest: "{{ ansible_cfg_inventory }}" + force: yes + trim_blocks: True + lstrip_blocks: True + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_fio_tests + - ansible_hosts_template.stat.exists + - name: Infer enabled mmtests test types set_fact: mmtests_enabled_test_types: >-
playbooks/roles/gen_hosts/templates/fio-tests.j2 new file +28 -0
@@ -0,0 +1,28 @@ +[all] +localhost ansible_connection=local +{{ kdevops_host_prefix }}-fio-tests +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-fio-tests-dev +{% endif %} +[all:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +[baseline] +{{ kdevops_host_prefix }}-fio-tests +[baseline:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +[dev] +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-fio-tests-dev +{% endif %} +[dev:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +[fio_tests] +{{ kdevops_host_prefix }}-fio-tests +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-fio-tests-dev +{% endif %} +[fio_tests:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +[service] +[service:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/hosts.j2 +38 -0
@@ -39,6 +39,44 @@ ansible_python_interpreter = "{{ kdevops_python_interpreter }}" [reboot-limit:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +{% elif kdevops_workflow_enable_fio_tests %} +[all] +localhost ansible_connection=local +{{ kdevops_host_prefix }}-fio-tests +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-fio-tests-dev +{% endif %} + +[all:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +[baseline] +{{ kdevops_host_prefix }}-fio-tests + +[baseline:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% if kdevops_baseline_and_dev %} +[dev] +{{ kdevops_host_prefix }}-fio-tests-dev + +[dev:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +{% endif %} +[fio_tests] +{{ kdevops_host_prefix }}-fio-tests +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-fio-tests-dev +{% endif %} + +[fio_tests:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" + +[service] + +[service:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}" {% else %} [all] localhost ansible_connection=local
playbooks/roles/gen_nodes/tasks/main.yml +32 -0
@@ -508,6 +508,38 @@ - kdevops_workflow_enable_sysbench - ansible_nodes_template.stat.exists +- name: Generate the fio-tests kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template + tags: [ 'hosts' ] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + nodes: "{{ [kdevops_host_prefix + '-fio-tests'] }}" + all_generic_nodes: "{{ [kdevops_host_prefix + '-fio-tests'] }}" + template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: yes + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_fio_tests + - ansible_nodes_template.stat.exists + - not kdevops_baseline_and_dev + +- name: Generate the fio-tests kdevops nodes file with dev hosts using {{ kdevops_nodes_template }} as jinja2 source template + tags: [ 'hosts' ] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + nodes: "{{ [kdevops_host_prefix + '-fio-tests', kdevops_host_prefix + '-fio-tests-dev'] }}" + all_generic_nodes: "{{ [kdevops_host_prefix + '-fio-tests', kdevops_host_prefix + '-fio-tests-dev'] }}" + template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: yes + when: + - kdevops_workflows_dedicated_workflow + - kdevops_workflow_enable_fio_tests + - ansible_nodes_template.stat.exists + - kdevops_baseline_and_dev + - name: Infer enabled mmtests test section types set_fact: mmtests_enabled_test_types: >-
workflows/Makefile +4 -0
@@ -62,6 +62,10 @@ ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_ENABLE_SSD_STEADY_STATE)) include workflows/steady_state/Makefile endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_SSD_STEADY_STATE == y +ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS)) +include workflows/fio-tests/Makefile +endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS == y + ANSIBLE_EXTRA_ARGS += $(WORKFLOW_ARGS) ANSIBLE_EXTRA_ARGS_SEPARATED += $(WORKFLOW_ARGS_SEPARATED) ANSIBLE_EXTRA_ARGS_DIRECT += $(WORKFLOW_ARGS_DIRECT)
workflows/fio-tests/Kconfig new file +420 -0
420 lines not on this page · raw message
workflows/fio-tests/Makefile new file +68 -0
47 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.45.2
Part of the diff is not shown here. View the raw message.
Quoted 79 lines · Luis Chamberlain
I had written fio-tests [0] a long time ago but although it used
Kconfig, at that point I had lacked the for-sight of leveraging jinja2 / ansible to make things more declarative. It's been on my back log to try to get that ported over to kdevops but now with Claude Code, it just required a series of prompts. And its even better now. *This* is how we scale.I've added a demo tree which just has the graphs for those just itching to see what this produces [1]. It's just a demo comparing two separate runs don't get too excited as its just a silly guest with virtio drives. However this should hopefully show you how easily and quickly with the new A/B testing feature.
The run time for tests is configurable, you also use the
FIO_QUICKenvironment variable to do quick tests.The runtime choices are:
- Default: 60 seconds runtime, 10 seconds ramp time
- Quick: 10 seconds runtime, 2 seconds ramp time (selected with FIO_QUICK=y)
- Custom High: 300 seconds runtime, 30 seconds ramp time
- Custom Low: 5 seconds runtime, 1 second ramp timeWe add defconfig-fio-tests-perf which to enable all performance testing knobs:
- All block sizes (4K, 8K, 16K, 32K, 64K, 128K)
- All IO depths (1, 4, 8, 16, 32, 64)
- All job counts (1, 2, 4, 8, 16)
- All test patterns (random/sequential read/write, mixed workloads)
- High DPI (300) for graphs
- A/B testing with baseline and dev nodesThis allows quick CI testing with:
make defconfig-fio-tests-perf FIO_QUICK=yOr comprehensive performance testing with:
make defconfig-fio-tests-perfKey features:
- Configurable test matrix: block sizes (4K-128K), IO depths (1-64), job counts
- Multiple workload patterns: random/sequential read/write, mixed workloads
- Advanced configuration: IO engines, direct IO, fsync options
- Performance logging: bandwidth, IOPS, and latency metrics
- Baseline management and results analysis
-FIO_TESTS_ENABLE_GRAPHING: Enable/disable graphing capabilities
- Graph format, DPI, and theme configuration options
- Updated CI defconfig with graphing support (150 DPI for faster CI)Key graphing features support:
- Performance analysis: bandwidth heatmaps, IOPS scaling, latency distributions
- A/B comparison: baseline vs development configuration analysis
- Trend analysis: block size scaling, IO depth optimization, correlation matrices
- Configurable output: PNG/SVG/PDF formats, DPI settings, matplotlib themesDocumentation:
-docs/fio-tests.md: Comprehensive workflow documentation covering:
* Origin story and relationship to upstream fio-tests project
* Quick start and configuration examples
* Test matrix configuration and device setup
* A/B testing and baseline management
* Graphing and visualization capabilities
* CI integration and troubleshooting guides
* Best practices and performance considerationsA minimal CI configuration (defconfig-fio-tests-ci) enables automated testing in GitHub Actions using
/dev/nullas the target device with a reduced test matrix for fast execution.We extend PROMPTS.md with prompts used for all this.
Usage:
make defconfig-fio-tests-ci # Simple CI testing make menuconfig # Interactive configuration make fio-tests # Run performance tests make fio-tests-baseline # Establish baseline make fio-tests-results # Collect results make fio-tests-graph # Generate performance graphs make fio-tests-compare # Compare baseline vs dev results make fio-tests-trend-analysis # Analyze performance trends
Quoted 4 lines
Link https://github.com/mcgrof/fio-tests # [0]Generated-by Claude AISigned-off-by Luis Chamberlain <[email protected]>
I've rebased and fixed ansible-lint issues and pushed.
Luis
Quoted 7 lines · Luis Chamberlain
workflows/fio-tests/Makefile new file +1
-0@@ -0,0 +1,68 @@ +ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
workflows/fio-tests/Makefile line 11
Quoted 10 lines
workflows/fio-tests/Makefile new file +10
-0+export KDEVOPS_HOSTS_TEMPLATE := fio-tests.j2 +endif + +fio-tests: + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ + --connection=ssh \ + --inventory hosts \ + --extra-vars=@$(KDEVOPS_EXTRA_VARS) \ + playbooks/fio-tests.yml \ + $(LIMIT_HOSTS)
I missed this...
The --connection and --inventory arguments shouldn't be necessary. They are managed by the inventory (hosts) file itself and the ansible.cfg.
Can you add a cleanup patch on top so it's consistent with the rest of the wrappers?
FYI, in future cleanups, I'd also like to get rid of the ANSIBLE_VERBOSE and control it through ansible.cfg (key: vebosity) [1] and expose it as kconfig option.
This should apply to all the ansible-playbook wrappers added.
Quoted 9 lines
workflows/fio-tests/Makefile new file +9
-0+ +fio-tests-baseline: + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ + --connection=ssh \ + --inventory hosts \ + --extra-vars=@$(KDEVOPS_EXTRA_VARS) \ + playbooks/fio-tests-baseline.yml \ + $(LIMIT_HOSTS) +
...
Quoted 21 lines · Daniel Gomez
Quoted 7 lines · Luis Chamberlain
workflows/fio-tests/Makefile new file +1
-0@@ -0,0 +1,68 @@ +ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))workflows/fio-tests/Makefile line 11
Quoted 10 lines
workflows/fio-tests/Makefile new file +10
-0+export KDEVOPS_HOSTS_TEMPLATE := fio-tests.j2 +endif + +fio-tests: + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ + --connection=ssh \ + --inventory hosts \ + --extra-vars=@$(KDEVOPS_EXTRA_VARS) \ + playbooks/fio-tests.yml \ + $(LIMIT_HOSTS)I missed this...
Quoted 5 lines
The --connection and --inventory arguments shouldn't be necessary. They are managed by the inventory (hosts) file itself and the ansible.cfg.
Can you add a cleanup patch on top so it's consistent with the rest of the wrappers?
Pushed.
Quoted 7 lines
FYI, in future cleanups, I'd also like to get rid of the
ANSIBLE_VERBOSEand control it through ansible.cfg (key: vebosity) [1] and expose it as kconfig option.This should apply to all the ansible-playbook wrappers added.
OK, well until then I'll keep it for now.
Luis