Skip to contentSkip to conversation

workflows: add fio-tests performance tests

PATCH In review Luis Chamberlain 2 participants 4 messages Aug 21, 2025 – Aug 26, 2025

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 workflows: add fio-tests performance tests

I had written fio-tests [0] a long time ago but although it used Kconfig, at that point I had lacked the for-sight of leveraging jinja2 / ansible to make things more declarative. It's been on my back log to try to get that ported over to kdevops but now with Claude Code, it just required a series of prompts. And its even better now. *This* is how we scale.

I've added a demo tree which just has the graphs for those just itching to see what this produces [1]. It's just a demo comparing two separate runs don't get too excited as its just a silly guest with virtio drives. However this should hopefully show you how easily and quickly with the new A/B testing feature.

The run time for tests is configurable, you also use the FIO_QUICK environment variable to do quick tests.

The runtime choices are:
- Default: 60 seconds runtime, 10 seconds ramp time
- Quick: 10 seconds runtime, 2 seconds ramp time (selected with FIO_QUICK=y)
- Custom High: 300 seconds runtime, 30 seconds ramp time
- Custom Low: 5 seconds runtime, 1 second ramp time

We add defconfig-fio-tests-perf which to enable all performance testing knobs:
- All block sizes (4K, 8K, 16K, 32K, 64K, 128K)
- All IO depths (1, 4, 8, 16, 32, 64)
- All job counts (1, 2, 4, 8, 16)
- All test patterns (random/sequential read/write, mixed workloads)
- High DPI (300) for graphs
- A/B testing with baseline and dev nodes

This allows quick CI testing with:
make defconfig-fio-tests-perf FIO_QUICK=y

Or comprehensive performance testing with:
make defconfig-fio-tests-perf

Key features:
- Configurable test matrix: block sizes (4K-128K), IO depths (1-64), job counts
- Multiple workload patterns: random/sequential read/write, mixed workloads
- Advanced configuration: IO engines, direct IO, fsync options
- Performance logging: bandwidth, IOPS, and latency metrics
- Baseline management and results analysis
- FIO_TESTS_ENABLE_GRAPHING: Enable/disable graphing capabilities
- Graph format, DPI, and theme configuration options
- Updated CI defconfig with graphing support (150 DPI for faster CI)

Key graphing features support:
- Performance analysis: bandwidth heatmaps, IOPS scaling, latency distributions
- A/B comparison: baseline vs development configuration analysis
- Trend analysis: block size scaling, IO depth optimization, correlation matrices
- Configurable output: PNG/SVG/PDF formats, DPI settings, matplotlib themes

Documentation:
- docs/fio-tests.md: Comprehensive workflow documentation covering:
* Origin story and relationship to upstream fio-tests project
* Quick start and configuration examples
* Test matrix configuration and device setup
* A/B testing and baseline management
* Graphing and visualization capabilities
* CI integration and troubleshooting guides
* Best practices and performance considerations

A minimal CI configuration (defconfig-fio-tests-ci) enables automated testing in GitHub Actions using /dev/null as the target device with a reduced test matrix for fast execution.

We extend PROMPTS.md with prompts used for all this.

Usage:

  make defconfig-fio-tests-ci    # Simple CI testing
  make menuconfig                # Interactive configuration
  make fio-tests                 # Run performance tests
  make fio-tests-baseline        # Establish baseline
  make fio-tests-results         # Collect results
  make fio-tests-graph           # Generate performance graphs
  make fio-tests-compare         # Compare baseline vs dev results
  make fio-tests-trend-analysis  # Analyze performance trends

---

Changes on this v2:

- Rebased
- The last graphs didn't make too much sense so I've asked for some new ones. I think we can start with this for now.
- Removed the docker tests as we can't run make menuconfig on a docker container. We can later enable on bare metal testing with the defconfigs/fio-tests-ci. It should be fast.

 .gitignore                                    |   2 +
 PROMPTS.md                                    | 105 ++++
 README.md                                     |  10 +
 defconfigs/fio-tests-ci                       |  56 ++
 defconfigs/fio-tests-perf                     |  58 +++
 docs/fio-tests.md                             | 377 ++++++++++++++
 kconfigs/workflows/Kconfig                    |  27 +
 playbooks/fio-tests-baseline.yml              |  29 ++
 playbooks/fio-tests-compare.yml               |  33 ++
 playbooks/fio-tests-graph.yml                 |  78 +++
 playbooks/fio-tests-results.yml               |  10 +
 playbooks/fio-tests-trend-analysis.yml        |  30 ++
 playbooks/fio-tests.yml                       |  11 +
 .../python/workflows/fio-tests/fio-compare.py | 383 ++++++++++++++
 .../python/workflows/fio-tests/fio-plot.py    | 350 +++++++++++++
 .../workflows/fio-tests/fio-trend-analysis.py | 477 ++++++++++++++++++
 playbooks/roles/fio-tests/defaults/main.yml   |  48 ++
 .../tasks/install-deps/debian/main.yml        |  20 +
 .../fio-tests/tasks/install-deps/main.yml     |   3 +
 .../tasks/install-deps/redhat/main.yml        |  20 +
 .../tasks/install-deps/suse/main.yml          |  20 +
 playbooks/roles/fio-tests/tasks/main.yaml     | 170 +++++++
 .../roles/fio-tests/templates/fio-job.ini.j2  |  29 ++
 playbooks/roles/gen_hosts/tasks/main.yml      |  13 +
 .../roles/gen_hosts/templates/fio-tests.j2    |  28 +
 playbooks/roles/gen_hosts/templates/hosts.j2  |  38 ++
 playbooks/roles/gen_nodes/tasks/main.yml      |  32 ++
 workflows/Makefile                            |   4 +
 workflows/fio-tests/Kconfig                   | 420 +++++++++++++++
 workflows/fio-tests/Makefile                  |  68 +++
 30 files changed, 2949 insertions(+)
 create mode 100644 defconfigs/fio-tests-ci
 create mode 100644 defconfigs/fio-tests-perf
 create mode 100644 docs/fio-tests.md
 create mode 100644 playbooks/fio-tests-baseline.yml
 create mode 100644 playbooks/fio-tests-compare.yml
 create mode 100644 playbooks/fio-tests-graph.yml
 create mode 100644 playbooks/fio-tests-results.yml
 create mode 100644 playbooks/fio-tests-trend-analysis.yml
 create mode 100644 playbooks/fio-tests.yml
 create mode 100755 playbooks/python/workflows/fio-tests/fio-compare.py
 create mode 100755 playbooks/python/workflows/fio-tests/fio-plot.py
 create mode 100755 playbooks/python/workflows/fio-tests/fio-trend-analysis.py
 create mode 100644 playbooks/roles/fio-tests/defaults/main.yml
 create mode 100644 playbooks/roles/fio-tests/tasks/install-deps/debian/main.yml
 create mode 100644 playbooks/roles/fio-tests/tasks/install-deps/main.yml
 create mode 100644 playbooks/roles/fio-tests/tasks/install-deps/redhat/main.yml
 create mode 100644 playbooks/roles/fio-tests/tasks/install-deps/suse/main.yml
 create mode 100644 playbooks/roles/fio-tests/tasks/main.yaml
 create mode 100644 playbooks/roles/fio-tests/templates/fio-job.ini.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/fio-tests.j2
 create mode 100644 workflows/fio-tests/Kconfig
 create mode 100644 workflows/fio-tests/Makefile
.gitignore +2 -0
index cfafa909cb40..50dc877adff5 100644
@@ -70,6 +70,8 @@ workflows/sysbench/results/
 workflows/mmtests/results/
 tmp
 
+workflows/fio-tests/results/
+
 playbooks/roles/linux-mirror/linux-mirror-systemd/*.service
 playbooks/roles/linux-mirror/linux-mirror-systemd/*.timer
 playbooks/roles/linux-mirror/linux-mirror-systemd/mirrors.yaml
PROMPTS.md +105 -0

111 lines not on this page · raw message

README.md +10 -0
index c471277eac11..0c30762a269f 100644
@@ -13,6 +13,7 @@ Table of Contents
       * [CXL](#cxl)
       * [reboot-limit](#reboot-limit)
       * [sysbench](#sysbench)
+      * [fio-tests](#fio-tests)
    * [kdevops chats](#kdevops-chats)
    * [kdevops on discord](#kdevops-on-discord)
       * [kdevops IRC](#kdevops-irc)
@@ -263,6 +264,15 @@ kdevops supports automation of sysbench tests on VMs with or without
 providers. For details refer to the
 [kdevops sysbench documentation](docs/sysbench/sysbench.md).
 
+### fio-tests
+
+kdevops includes comprehensive storage performance testing through the fio-tests
+workflow, adapted from the original [fio-tests framework](https://github.com/mcgrof/fio-tests).
+This workflow provides flexible I/O benchmarking with configurable test matrices,
+A/B testing capabilities, and advanced graphing and visualization support. For
+detailed configuration and usage information, refer to the
+[kdevops fio-tests documentation](docs/fio-tests.md).
+
 ## kdevops chats
 
 We use discord and IRC. Right now we have more folks on discord than on IRC.
defconfigs/fio-tests-ci new file +56 -0

56 lines not on this page · raw message

defconfigs/fio-tests-perf new file +58 -0

58 lines not on this page · raw message

docs/fio-tests.md new file +377 -0

377 lines not on this page · raw message

kconfigs/workflows/Kconfig +27 -0
index b1b8a48b8536..6b2a37696afd 100644
@@ -207,6 +207,13 @@ config KDEVOPS_WORKFLOW_DEDICATE_MMTESTS
 	  This will dedicate your configuration to running only the
 	  mmtests workflow for memory fragmentation testing.
 
+config KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS
+	bool "fio-tests"
+	select KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS
+	help
+	  This will dedicate your configuration to running only the
+	  fio-tests workflow for comprehensive storage performance testing.
+
 endchoice
 
 config KDEVOPS_WORKFLOW_NAME
@@ -221,6 +228,7 @@ config KDEVOPS_WORKFLOW_NAME
 	default "nfstest" if KDEVOPS_WORKFLOW_DEDICATE_NFSTEST
 	default "sysbench" if KDEVOPS_WORKFLOW_DEDICATE_SYSBENCH
 	default "mmtests" if KDEVOPS_WORKFLOW_DEDICATE_MMTESTS
+	default "fio-tests" if KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS
 
 endif
 
@@ -322,6 +330,14 @@ config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_MMTESTS
 	  Select this option if you want to provision mmtests on a
 	  single target node for by-hand testing.
 
+config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_FIO_TESTS
+	bool "fio-tests"
+	select KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS
+	depends on LIBVIRT || TERRAFORM_PRIVATE_NET
+	help
+	  Select this option if you want to provision fio-tests on a
+	  single target node for by-hand testing.
+
 endif # !WORKFLOWS_DEDICATED_WORKFLOW
 
 config KDEVOPS_WORKFLOW_ENABLE_FSTESTS
@@ -435,6 +451,17 @@ source "workflows/mmtests/Kconfig"
 endmenu
 endif # KDEVOPS_WORKFLOW_ENABLE_MMTESTS
 
+config KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS
+	bool
+	output yaml
+	default y if KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_FIO_TESTS || KDEVOPS_WORKFLOW_DEDICATE_FIO_TESTS
+
+if KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS
+menu "Configure and run fio-tests"
+source "workflows/fio-tests/Kconfig"
+endmenu
+endif # KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS
+
 config KDEVOPS_WORKFLOW_ENABLE_SSD_STEADY_STATE
        bool "Attain SSD steady state prior to tests"
        output yaml
playbooks/fio-tests-baseline.yml new file +29 -0
new file mode 100644
index 000000000000..1f990c0efa40
@@ -0,0 +1,29 @@
+---
+- hosts:
+    - baseline
+    - dev
+  become: no
+  vars:
+    ansible_ssh_pipelining: True
+  tasks:
+    - name: Create baseline directory structure
+      file:
+        path: "{{ fio_tests_results_dir }}/baseline"
+        state: directory
+        mode: '0755'
+      become: yes
+
+    - name: Save current test configuration as baseline
+      copy:
+        src: "{{ fio_tests_results_dir }}/results_{{ item }}.txt"
+        dest: "{{ fio_tests_results_dir }}/baseline/baseline_{{ item }}.txt"
+        remote_src: yes
+        backup: yes
+      with_fileglob:
+        - "{{ fio_tests_results_dir }}/results_*.txt"
+      become: yes
+      ignore_errors: yes
+
+    - name: Create baseline timestamp
+      shell: date > "{{ fio_tests_results_dir }}/baseline/baseline_timestamp.txt"
+      become: yes
playbooks/fio-tests-compare.yml new file +33 -0
new file mode 100644
index 000000000000..e6e3464613c2
@@ -0,0 +1,33 @@
+---
+- hosts: localhost
+  become: no
+  vars:
+    ansible_ssh_pipelining: True
+  tasks:
+    - name: Check if baseline and dev hosts exist in inventory
+      fail:
+        msg: "Both baseline and dev hosts must exist for comparison"
+      when: "'baseline' not in groups or 'dev' not in groups"
+
+    - name: Create local graph results comparison directory
+      file:
+        path: "{{ topdir_path }}/workflows/fio-tests/results/graphs"
+        state: directory
+        mode: '0755'
+
+    - name: Generate comparison graphs
+      shell: |
+        python3 {{ topdir_path }}/playbooks/python/workflows/fio-tests/fio-compare.py \
+          {{ topdir_path }}/workflows/fio-tests/results/{{ groups['baseline'][0] }}/fio-tests-results-{{ groups['baseline'][0] }}   \
+          {{ topdir_path }}/workflows/fio-tests/results/{{ groups['dev'][0] }}/fio-tests-results-{{ groups['dev'][0] }}        \
+          --output-dir {{ topdir_path }}/workflows/fio-tests/results/graphs           \
+          --baseline-label "Baseline"                                                 \
+          --dev-label "Development"
+
+    - name: List comparison results
+      shell: ls -la {{ topdir_path }}/workflows/fio-tests/results/graphs
+      register: comparison_list
+
+    - name: Display comparison results
+      debug:
+        msg: "{{ comparison_list.stdout_lines }}"
playbooks/fio-tests-graph.yml new file +78 -0

78 lines not on this page · raw message

playbooks/fio-tests-results.yml new file +10 -0
new file mode 100644
index 000000000000..dcc2ea1847e9
@@ -0,0 +1,10 @@
+---
+- hosts:
+    - baseline
+    - dev
+  become: no
+  vars:
+    ansible_ssh_pipelining: True
+  roles:
+    - role: fio-tests
+      tags: ['results']
playbooks/fio-tests-trend-analysis.yml new file +30 -0
new file mode 100644
index 000000000000..e94a1a1f76ee
@@ -0,0 +1,30 @@
+---
+- hosts:
+    - baseline
+    - dev
+  become: no
+  vars:
+    ansible_ssh_pipelining: True
+  tasks:
+    - include_role:
+        name: create_data_partition
+      tags: [ 'oscheck', 'data_partition' ]
+
+    - name: Generate fio trend analysis
+      shell: |
+        cd {{ fio_tests_results_dir }}
+        mkdir -p analysis
+        python3 {{ kdevops_data }}/playbooks/python/workflows/fio-tests/fio-trend-analysis.py \
+          . --output-dir analysis
+      args:
+        creates: "{{ fio_tests_results_dir }}/analysis/block_size_trends.png"
+      become: yes
+
+    - name: List generated analysis files
+      shell: ls -la {{ fio_tests_results_dir }}/analysis/
+      become: yes
+      register: analysis_list
+
+    - name: Display generated analysis files
+      debug:
+        msg: "{{ analysis_list.stdout_lines }}"
playbooks/fio-tests.yml new file +11 -0
new file mode 100644
index 000000000000..b9f5f936c653
@@ -0,0 +1,11 @@
+---
+- hosts:
+    - baseline
+    - dev
+  become: no
+  vars:
+    ansible_ssh_pipelining: True
+  roles:
+    - role: create_data_partition
+      tags: ['data_partition']
+    - role: fio-tests
playbooks/python/workflows/fio-tests/fio-compare.py new file +383 -0

383 lines not on this page · raw message

playbooks/python/workflows/fio-tests/fio-plot.py new file +350 -0

350 lines not on this page · raw message

playbooks/python/workflows/fio-tests/fio-trend-analysis.py new file +477 -0

477 lines not on this page · raw message

playbooks/roles/fio-tests/defaults/main.yml new file +48 -0

48 lines not on this page · raw message

playbooks/roles/fio-tests/tasks/install-deps/debian/main.yml new file +20 -0
new file mode 100644
index 000000000000..327ab493ba81
@@ -0,0 +1,20 @@
+---
+- name: Install fio for Debian/Ubuntu
+  package:
+    name:
+      - fio
+      - python3
+    state: present
+  become: yes
+
+- name: Install graphing dependencies for Debian/Ubuntu
+  package:
+    name:
+      - python3-pip
+      - python3-pandas
+      - python3-matplotlib
+      - python3-seaborn
+      - python3-numpy
+    state: present
+  become: yes
+  when: fio_tests_enable_graphing is defined and fio_tests_enable_graphing
playbooks/roles/fio-tests/tasks/install-deps/main.yml new file +3 -0
new file mode 100644
index 000000000000..c29e6f751fce
@@ -0,0 +1,3 @@
+---
+- name: Include distribution-specific installation tasks
+  include_tasks: "{{ ansible_os_family | lower }}/main.yml"
playbooks/roles/fio-tests/tasks/install-deps/redhat/main.yml new file +20 -0
new file mode 100644
index 000000000000..7decdcf84fc0
@@ -0,0 +1,20 @@
+---
+- name: Install fio for RHEL/CentOS/Fedora
+  package:
+    name:
+      - fio
+      - python3
+    state: present
+  become: yes
+
+- name: Install graphing dependencies for RHEL/CentOS/Fedora
+  package:
+    name:
+      - python3-pip
+      - python3-pandas
+      - python3-matplotlib
+      - python3-seaborn
+      - python3-numpy
+    state: present
+  become: yes
+  when: fio_tests_enable_graphing is defined and fio_tests_enable_graphing
playbooks/roles/fio-tests/tasks/install-deps/suse/main.yml new file +20 -0
new file mode 100644
index 000000000000..8bd5cf6261b2
@@ -0,0 +1,20 @@
+---
+- name: Install fio for SUSE
+  package:
+    name:
+      - fio
+      - python3
+    state: present
+  become: yes
+
+- name: Install graphing dependencies for SUSE
+  package:
+    name:
+      - python3-pip
+      - python3-pandas
+      - python3-matplotlib
+      - python3-seaborn
+      - python3-numpy
+    state: present
+  become: yes
+  when: fio_tests_enable_graphing is defined and fio_tests_enable_graphing
playbooks/roles/fio-tests/tasks/main.yaml new file +170 -0

170 lines not on this page · raw message

playbooks/roles/fio-tests/templates/fio-job.ini.j2 new file +29 -0
new file mode 100644
index 000000000000..49727d461e36
@@ -0,0 +1,29 @@
+[global]
+ioengine={{ fio_tests_ioengine }}
+{% if fio_tests_device == '/dev/null' %}
+direct=0
+{% else %}
+direct={{ fio_tests_direct | int }}
+{% endif %}
+{% if fio_tests_device == '/dev/null' %}
+fsync_on_close=0
+{% else %}
+fsync_on_close={{ fio_tests_fsync_on_close | int }}
+{% endif %}
+group_reporting=1
+time_based=1
+runtime={{ fio_tests_runtime }}
+ramp_time={{ fio_tests_ramp_time }}
+
+[{{ pattern.name }}_bs{{ block_size }}_iodepth{{ io_depth }}_jobs{{ num_jobs }}]
+filename={{ fio_tests_device }}
+{% if fio_tests_device == '/dev/null' %}
+size=1G
+{% endif %}
+rw={{ pattern.rw }}
+{% if pattern.rwmixread is defined and pattern.rw in ['randrw', 'rw'] %}
+rwmixread={{ pattern.rwmixread }}
+{% endif %}
+bs={{ block_size }}
+iodepth={{ io_depth }}
+numjobs={{ num_jobs }}
playbooks/roles/gen_hosts/tasks/main.yml +13 -0
index e36d71fc5535..febe81ff2a39 100644
@@ -324,6 +324,19 @@
     - kdevops_workflow_enable_sysbench
     - ansible_hosts_template.stat.exists
 
+- name: Generate the Ansible hosts file for a dedicated fio-tests setup
+  tags: [ 'hosts' ]
+  template:
+    src: "{{ kdevops_hosts_template }}"
+    dest: "{{ ansible_cfg_inventory }}"
+    force: yes
+    trim_blocks: True
+    lstrip_blocks: True
+  when:
+    - kdevops_workflows_dedicated_workflow
+    - kdevops_workflow_enable_fio_tests
+    - ansible_hosts_template.stat.exists
+
 - name: Infer enabled mmtests test types
   set_fact:
     mmtests_enabled_test_types: >-
playbooks/roles/gen_hosts/templates/fio-tests.j2 new file +28 -0
new file mode 100644
index 000000000000..75bc0c53569e
@@ -0,0 +1,28 @@
+[all]
+localhost ansible_connection=local
+{{ kdevops_host_prefix }}-fio-tests
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-fio-tests-dev
+{% endif %}
+[all:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+[baseline]
+{{ kdevops_host_prefix }}-fio-tests
+[baseline:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+[dev]
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-fio-tests-dev
+{% endif %}
+[dev:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+[fio_tests]
+{{ kdevops_host_prefix }}-fio-tests
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-fio-tests-dev
+{% endif %}
+[fio_tests:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+[service]
+[service:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/hosts.j2 +38 -0
index f89fae48c349..6d83191d93ce 100644
@@ -39,6 +39,44 @@ ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
 
 [reboot-limit:vars]
 ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+{% elif kdevops_workflow_enable_fio_tests %}
+[all]
+localhost ansible_connection=local
+{{ kdevops_host_prefix }}-fio-tests
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-fio-tests-dev
+{% endif %}
+
+[all:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+[baseline]
+{{ kdevops_host_prefix }}-fio-tests
+
+[baseline:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+{% if kdevops_baseline_and_dev %}
+[dev]
+{{ kdevops_host_prefix }}-fio-tests-dev
+
+[dev:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+{% endif %}
+[fio_tests]
+{{ kdevops_host_prefix }}-fio-tests
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-fio-tests-dev
+{% endif %}
+
+[fio_tests:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+[service]
+
+[service:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
 {% else %}
 [all]
 localhost ansible_connection=local
playbooks/roles/gen_nodes/tasks/main.yml +32 -0
index 8c1772546800..0451761fbf30 100644
@@ -508,6 +508,38 @@
     - kdevops_workflow_enable_sysbench
     - ansible_nodes_template.stat.exists
 
+- name: Generate the fio-tests kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template
+  tags: [ 'hosts' ]
+  vars:
+    node_template: "{{ kdevops_nodes_template | basename }}"
+    nodes: "{{ [kdevops_host_prefix + '-fio-tests'] }}"
+    all_generic_nodes: "{{ [kdevops_host_prefix + '-fio-tests'] }}"
+  template:
+    src: "{{ node_template }}"
+    dest: "{{ topdir_path }}/{{ kdevops_nodes }}"
+    force: yes
+  when:
+    - kdevops_workflows_dedicated_workflow
+    - kdevops_workflow_enable_fio_tests
+    - ansible_nodes_template.stat.exists
+    - not kdevops_baseline_and_dev
+
+- name: Generate the fio-tests kdevops nodes file with dev hosts using {{ kdevops_nodes_template }} as jinja2 source template
+  tags: [ 'hosts' ]
+  vars:
+    node_template: "{{ kdevops_nodes_template | basename }}"
+    nodes: "{{ [kdevops_host_prefix + '-fio-tests', kdevops_host_prefix + '-fio-tests-dev'] }}"
+    all_generic_nodes: "{{ [kdevops_host_prefix + '-fio-tests', kdevops_host_prefix + '-fio-tests-dev'] }}"
+  template:
+    src: "{{ node_template }}"
+    dest: "{{ topdir_path }}/{{ kdevops_nodes }}"
+    force: yes
+  when:
+    - kdevops_workflows_dedicated_workflow
+    - kdevops_workflow_enable_fio_tests
+    - ansible_nodes_template.stat.exists
+    - kdevops_baseline_and_dev
+
 - name: Infer enabled mmtests test section types
   set_fact:
     mmtests_enabled_test_types: >-
workflows/Makefile +4 -0
index ef3cc2d1f0ee..b5f54ff5f57b 100644
@@ -62,6 +62,10 @@ ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_ENABLE_SSD_STEADY_STATE))
 include workflows/steady_state/Makefile
 endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_SSD_STEADY_STATE == y
 
+ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS))
+include workflows/fio-tests/Makefile
+endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_FIO_TESTS == y
+
 ANSIBLE_EXTRA_ARGS += $(WORKFLOW_ARGS)
 ANSIBLE_EXTRA_ARGS_SEPARATED += $(WORKFLOW_ARGS_SEPARATED)
 ANSIBLE_EXTRA_ARGS_DIRECT += $(WORKFLOW_ARGS_DIRECT)
workflows/fio-tests/Kconfig new file +420 -0

420 lines not on this page · raw message

workflows/fio-tests/Makefile new file +68 -0

47 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.45.2

Part of the diff is not shown here. View the raw message.

#1 lore raw
Quoted 79 lines · Luis Chamberlain

I had written fio-tests [0] a long time ago but although it used Kconfig, at that point I had lacked the for-sight of leveraging jinja2 / ansible to make things more declarative. It's been on my back log to try to get that ported over to kdevops but now with Claude Code, it just required a series of prompts. And its even better now. *This* is how we scale.

I've added a demo tree which just has the graphs for those just itching to see what this produces [1]. It's just a demo comparing two separate runs don't get too excited as its just a silly guest with virtio drives. However this should hopefully show you how easily and quickly with the new A/B testing feature.

The run time for tests is configurable, you also use the FIO_QUICK environment variable to do quick tests.

The runtime choices are:
- Default: 60 seconds runtime, 10 seconds ramp time
- Quick: 10 seconds runtime, 2 seconds ramp time (selected with FIO_QUICK=y)
- Custom High: 300 seconds runtime, 30 seconds ramp time
- Custom Low: 5 seconds runtime, 1 second ramp time

We add defconfig-fio-tests-perf which to enable all performance testing knobs:
- All block sizes (4K, 8K, 16K, 32K, 64K, 128K)
- All IO depths (1, 4, 8, 16, 32, 64)
- All job counts (1, 2, 4, 8, 16)
- All test patterns (random/sequential read/write, mixed workloads)
- High DPI (300) for graphs
- A/B testing with baseline and dev nodes

This allows quick CI testing with:
make defconfig-fio-tests-perf FIO_QUICK=y

Or comprehensive performance testing with:
make defconfig-fio-tests-perf

Key features:
- Configurable test matrix: block sizes (4K-128K), IO depths (1-64), job counts
- Multiple workload patterns: random/sequential read/write, mixed workloads
- Advanced configuration: IO engines, direct IO, fsync options
- Performance logging: bandwidth, IOPS, and latency metrics
- Baseline management and results analysis
- FIO_TESTS_ENABLE_GRAPHING: Enable/disable graphing capabilities
- Graph format, DPI, and theme configuration options
- Updated CI defconfig with graphing support (150 DPI for faster CI)

Key graphing features support:
- Performance analysis: bandwidth heatmaps, IOPS scaling, latency distributions
- A/B comparison: baseline vs development configuration analysis
- Trend analysis: block size scaling, IO depth optimization, correlation matrices
- Configurable output: PNG/SVG/PDF formats, DPI settings, matplotlib themes

Documentation:
- docs/fio-tests.md: Comprehensive workflow documentation covering:
* Origin story and relationship to upstream fio-tests project
* Quick start and configuration examples
* Test matrix configuration and device setup
* A/B testing and baseline management
* Graphing and visualization capabilities
* CI integration and troubleshooting guides
* Best practices and performance considerations

A minimal CI configuration (defconfig-fio-tests-ci) enables automated testing in GitHub Actions using /dev/null as the target device with a reduced test matrix for fast execution.

We extend PROMPTS.md with prompts used for all this.

Usage:

  make defconfig-fio-tests-ci    # Simple CI testing
  make menuconfig                # Interactive configuration
  make fio-tests                 # Run performance tests
  make fio-tests-baseline        # Establish baseline
  make fio-tests-results         # Collect results
  make fio-tests-graph           # Generate performance graphs
  make fio-tests-compare         # Compare baseline vs dev results
  make fio-tests-trend-analysis  # Analyze performance trends
Quoted 4 lines

I've rebased and fixed ansible-lint issues and pushed.

Luis

#2 lore raw
In reply to Luis Chamberlain
Quoted 7 lines · Luis Chamberlain
workflows/fio-tests/Makefile new file +1 -0
new file mode 100644
index 000000000000..cc641606e98c
@@ -0,0 +1,68 @@
+ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))

workflows/fio-tests/Makefile line 11

Quoted 10 lines
workflows/fio-tests/Makefile new file +10 -0
+export KDEVOPS_HOSTS_TEMPLATE := fio-tests.j2
+endif
+
+fio-tests:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+		--connection=ssh \
+		--inventory hosts \
+		--extra-vars=@$(KDEVOPS_EXTRA_VARS) \
+		playbooks/fio-tests.yml \
+		$(LIMIT_HOSTS)

I missed this...

The --connection and --inventory arguments shouldn't be necessary. They are managed by the inventory (hosts) file itself and the ansible.cfg.

Can you add a cleanup patch on top so it's consistent with the rest of the wrappers?

FYI, in future cleanups, I'd also like to get rid of the ANSIBLE_VERBOSE and control it through ansible.cfg (key: vebosity) [1] and expose it as kconfig option.

This should apply to all the ansible-playbook wrappers added.

Quoted 9 lines
workflows/fio-tests/Makefile new file +9 -0
+
+fio-tests-baseline:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+		--connection=ssh \
+		--inventory hosts \
+		--extra-vars=@$(KDEVOPS_EXTRA_VARS) \
+		playbooks/fio-tests-baseline.yml \
+		$(LIMIT_HOSTS)
+

...

#3 lore raw
Quoted 21 lines · Daniel Gomez
Quoted 7 lines · Luis Chamberlain
workflows/fio-tests/Makefile new file +1 -0
new file mode 100644
index 000000000000..cc641606e98c
@@ -0,0 +1,68 @@
+ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))

workflows/fio-tests/Makefile line 11

Quoted 10 lines
workflows/fio-tests/Makefile new file +10 -0
+export KDEVOPS_HOSTS_TEMPLATE := fio-tests.j2
+endif
+
+fio-tests:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+		--connection=ssh \
+		--inventory hosts \
+		--extra-vars=@$(KDEVOPS_EXTRA_VARS) \
+		playbooks/fio-tests.yml \
+		$(LIMIT_HOSTS)

I missed this...

Quoted 5 lines

The --connection and --inventory arguments shouldn't be necessary. They are managed by the inventory (hosts) file itself and the ansible.cfg.

Can you add a cleanup patch on top so it's consistent with the rest of the wrappers?

Pushed.

Quoted 7 lines

FYI, in future cleanups, I'd also like to get rid of the ANSIBLE_VERBOSE and control it through ansible.cfg (key: vebosity) [1] and expose it as kconfig option.

This should apply to all the ansible-playbook wrappers added.

OK, well until then I'll keep it for now.

Luis