linux-ab enhancements + monitor support
cover linux-ab enhancements + monitor support
These are a collection of practical enhancements developed as I tested Linux AB testing with parallel writeback developmental patches to help do analysis on impact on memory management. The last patch adds the ability to do experimental (non upstream) monitoring of different types. We could obviously later add monitoring for existing upstream knobs but in this case we want to track folio migration success rates and so are using a currently out of tree knob.
Luis Chamberlain (8):
bootlinux: use different kernel for A/B testing by default
bootlinux: add support for custom refs on dev kernels on the CLI
bootlinux: add git ref verification before cloning
bootlinux: add git dirty check before cloning
bootlinux: add intelligent git repository detection and management
bootlinux: enhance A/B testing and repository management
fstests: add make target for running tests on all hosts
monitoring: integrate monitoring collection into fstests workflow
Kconfig | 4 + README.md | 1 + defconfigs/xfs_reflink_lbs | 1 + kconfigs/Kconfig.kdevops | 8 +- playbooks/roles/bootlinux/tasks/build/9p.yml | 193 +++++++++++++++++- .../roles/bootlinux/tasks/build/builder.yml | 165 +++++++++++++++ .../roles/bootlinux/tasks/build/targets.yml | 166 +++++++++++++++ playbooks/roles/fstests/tasks/main.yml | 14 ++ scripts/ensure_newlines.py | 3 +- workflows/fstests/Makefile | 10 + workflows/linux/Kconfig | 18 +- 11 files changed, 571 insertions(+), 12 deletions(-)
Signature · 1 line
2.47.2
PATCH 1/8 bootlinux: use different kernel for A/B testing by default
If you are doing A/B testing then by default you'd want to test different kernels typically. So enable that by default. This will help make our CIs easier to write leveraging defconfigs with less crap in them.
--- workflows/linux/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
workflows/linux/Kconfig +1 -1
@@ -385,7 +385,7 @@ if KDEVOPS_BASELINE_AND_DEV choice prompt "A/B kernel testing configuration" - default BOOTLINUX_AB_SAME_REF + default BOOTLINUX_AB_DIFFERENT_REF help When A/B testing is enabled, you can choose to use the same kernel reference for both baseline and dev nodes, or specify
Signature · 1 line
2.47.2
PATCH 2/8 bootlinux: add support for custom refs on dev kernels on the CLI
In reply to Luis ChamberlainWe already have support to easily test arbitrary kernels and refs by just using the LINUX_TREE and LINUX_REF environment variables. We leverage this for our CIs:
* LINUX_TREE: the target tree for baseline group (A)
* LINUX_REF: the target tree ref the baseline group (A)
Now that we have AB-testing support we want to first add support so that a simple TEST_AB=y will enable KDEVOPS_BASELINE_AND_DEV and then we want to be able to customize the target dev tree and ref. We do this with two other environment variables:
* LINUX_DEV_TREE: the target tree for dev group (B)
* LINUX_DEV_REF: the target tree ref the dev group (B)
You just need to make sure to also pass TEST_AB=y.
By leveraging this we can easily use AB testing in the future on CIs to compare and contrast different kernel releases against any target workload we have.
--- kconfigs/Kconfig.kdevops | 8 +++++++- workflows/linux/Kconfig | 16 ++++++++++++++-- 2 files changed, 21 insertions(+), 3 deletions(-)
kconfigs/Kconfig.kdevops +7 -1
@@ -51,6 +51,11 @@ config KDEVOPS_HOSTS_PREFIX_SET_BY_CLI bool default $(shell, scripts/check-cli-set-var.sh KDEVOPS_HOSTS_PREFIX) +config KDEVOPS_BASELINE_AND_DEV_SET_BY_CLI + bool + output yaml + default $(shell, scripts/check-cli-set-var.sh TEST_AB) + config KDEVOPS_HOSTS_PREFIX string "The hostname prefix to use for nodes" output yaml
@@ -78,7 +83,8 @@ config KDEVOPS_CUSTOM_SSH_KEXALGORITHMS config KDEVOPS_BASELINE_AND_DEV bool "Enable both a baseline and development system per target test" - default n + default n if !KDEVOPS_BASELINE_AND_DEV_SET_BY_CLI + default y if KDEVOPS_BASELINE_AND_DEV_SET_BY_CLI help By default kdevops will only spawn a baseline target node (local virtualization or cloud node) for your Linux kernel testing and
workflows/linux/Kconfig +14 -2
@@ -10,6 +10,16 @@ config BOOTLINUX_TREE_REF_SET_BY_CLI output yaml default $(shell, scripts/check-cli-set-var.sh LINUX_TREE_REF) +config BOOTLINUX_DEV_TREE_SET_BY_CLI + bool + output yaml + default $(shell, scripts/check-cli-set-var.sh LINUX_DEV_TREE) + +config BOOTLINUX_DEV_TREE_REF_SET_BY_CLI + bool + output yaml + default $(shell, scripts/check-cli-set-var.sh LINUX_DEV_TREE_REF) + config BOOTLINUX_HAS_PURE_IOMAP_CONFIG bool
@@ -415,7 +425,8 @@ if BOOTLINUX_AB_DIFFERENT_REF config BOOTLINUX_DEV_TREE string "Development kernel tree URL" output yaml - default BOOTLINUX_TREE + default BOOTLINUX_TREE if !BOOTLINUX_DEV_TREE_SET_BY_CLI + default $(shell, ./scripts/append-makefile-vars.sh $(LINUX_DEV_TREE)) if BOOTLINUX_DEV_TREE_SET_BY_CLI help Git tree URL for the development kernel. If left empty or same as the baseline tree, the same tree will be used with a different
@@ -424,7 +435,8 @@ config BOOTLINUX_DEV_TREE config TARGET_LINUX_DEV_REF string "Development kernel reference" output yaml - default $(shell, scripts/infer_last_stable_kernel.sh) + default $(shell, scripts/infer_last_stable_kernel.sh) if !BOOTLINUX_DEV_TREE_REF_SET_BY_CLI + default $(shell, ./scripts/append-makefile-vars.sh $(LINUX_DEV_TREE_REF)) if BOOTLINUX_DEV_TREE_REF_SET_BY_CLI help Git reference (branch, tag, or commit) for the development kernel. This should be different from the baseline reference to enable
Signature · 1 line
2.47.2
PATCH 3/8 bootlinux: add git ref verification before cloning
In reply to Luis ChamberlainAdd preliminary verification tasks to check if the target git ref exists before attempting to clone the Linux kernel repository. This prevents confusing git clone failures and provides clear, actionable error messages to users.
The verification is particularly important for A/B testing scenarios where:
- Different kernel refs may be used for baseline vs development builds
- Shallow clones might not contain all required refs
- Users may specify refs that don't exist in the repository
Each build method now verifies the ref availability:
- 9p.yml: Verifies active_linux_ref (or target_linux_ref fallback)
- targets.yml: Verifies target_linux_ref on target nodes
- builder.yml: Verifies target_linux_ref on builder nodes
The error messages guide users to:
1. Check if the ref actually exists in the repository
2. Disable shallow cloning when using A/B testing with different refs
3. Verify the repository URL is correct and accessible
This change improves the user experience by failing fast with helpful diagnostics rather than letting git clone fail with cryptic errors.
--- playbooks/roles/bootlinux/tasks/build/9p.yml | 29 +++++++++++++++++++ .../roles/bootlinux/tasks/build/builder.yml | 21 ++++++++++++++ .../roles/bootlinux/tasks/build/targets.yml | 22 ++++++++++++++ 3 files changed, 72 insertions(+)
playbooks/roles/bootlinux/tasks/build/9p.yml +29 -0
@@ -44,6 +44,35 @@ - bootlinux_tree_set_by_cli|bool - not target_directory_stat.stat.exists +- name: Verify target git ref exists before cloning + command: "git ls-remote {{ target_linux_git }} {{ active_linux_ref | default(target_linux_ref) }}" + register: ref_check + run_once: true + delegate_to: localhost + when: + - not bootlinux_tree_set_by_cli|bool + tags: [ 'clone'] + +- name: Fail if git ref does not exist + fail: + msg: | + Failed to verify git ref '{{ active_linux_ref | default(target_linux_ref) }}' exists in repository '{{ target_linux_git }}'. + + This typically happens when: + 1. The ref (branch/tag/commit) doesn't exist in the repository + 2. You're using A/B testing with a shallow clone that doesn't contain the required ref + 3. The repository URL is incorrect or inaccessible + + Please verify: + - The ref '{{ active_linux_ref | default(target_linux_ref) }}' exists in the repository + - If using A/B testing with different refs, ensure shallow cloning is disabled + - The repository URL '{{ target_linux_git }}' is correct and accessible + when: + - not bootlinux_tree_set_by_cli|bool + - ref_check.rc != 0 + run_once: true + delegate_to: localhost + - name: git clone {{ target_linux_tree }} on the control node git: repo: "{{ target_linux_git }}"
playbooks/roles/bootlinux/tasks/build/builder.yml +21 -0
@@ -10,6 +10,27 @@ - target_linux_install_b4 - ansible_os_family == "Debian" +- name: Verify target git ref exists before cloning + command: "git ls-remote {{ target_linux_git }} {{ target_linux_ref }}" + register: ref_check + +- name: Fail if git ref does not exist + fail: + msg: | + Failed to verify git ref '{{ target_linux_ref }}' exists in repository '{{ target_linux_git }}'. + + This typically happens when: + 1. The ref (branch/tag/commit) doesn't exist in the repository + 2. You're using A/B testing with a shallow clone that doesn't contain the required ref + 3. The repository URL is incorrect or inaccessible + + Please verify: + - The ref '{{ target_linux_ref }}' exists in the repository + - If using A/B testing with different refs, ensure shallow cloning is disabled + - The repository URL '{{ target_linux_git }}' is correct and accessible + when: + - ref_check.rc != 0 + - name: Clone {{ target_linux_tree }} ansible.builtin.git: repo: "{{ target_linux_git }}"
playbooks/roles/bootlinux/tasks/build/targets.yml +22 -0
@@ -10,6 +10,28 @@ - target_linux_install_b4 - ansible_facts['os_family']|lower != 'debian' +- name: Verify target git ref exists before cloning + command: "git ls-remote {{ target_linux_git }} {{ target_linux_ref }}" + register: ref_check + tags: [ 'clone'] + +- name: Fail if git ref does not exist + fail: + msg: | + Failed to verify git ref '{{ target_linux_ref }}' exists in repository '{{ target_linux_git }}'. + + This typically happens when: + 1. The ref (branch/tag/commit) doesn't exist in the repository + 2. You're using A/B testing with a shallow clone that doesn't contain the required ref + 3. The repository URL is incorrect or inaccessible + + Please verify: + - The ref '{{ target_linux_ref }}' exists in the repository + - If using A/B testing with different refs, ensure shallow cloning is disabled + - The repository URL '{{ target_linux_git }}' is correct and accessible + when: + - ref_check.rc != 0 + - name: git clone {{ target_linux_tree }} on the target nodes git: repo: "{{ target_linux_git }}"
Signature · 1 line
2.47.2
PATCH 4/8 bootlinux: add git dirty check before cloning
In reply to Luis ChamberlainAdd preliminary checks to detect if the git tree has local modifications before attempting to clone or update. This provides clearer error messages when git operations fail due to uncommitted changes.
The issue was discovered when prior kdevops commits inadvertently modified files in the linux kernel tree through overly broad make style operations. When git clone/update encounters a dirty tree, it fails with a generic "Local modifications exist" error that doesn't guide users on resolution.
Each build method now checks for dirty trees:
- 9p.yml: Checks bootlinux_9p_host_path on control node
- targets.yml: Checks target_linux_dir_path on target nodes
- builder.yml: Checks target_linux_dir_path on builder nodes
The error messages provide clear resolution steps:
1. Commit or stash changes
2. Hard reset the tree
3. Remove the directory for a fresh clone
This change improves debugging by:
- Failing fast with actionable error messages
- Showing which files are modified
- Providing multiple resolution options
- Preventing confusing git errors downstream
--- playbooks/roles/bootlinux/tasks/build/9p.yml | 43 +++++++++++++++++++ .../roles/bootlinux/tasks/build/builder.yml | 33 ++++++++++++++ .../roles/bootlinux/tasks/build/targets.yml | 33 ++++++++++++++ 3 files changed, 109 insertions(+)
playbooks/roles/bootlinux/tasks/build/9p.yml +43 -0
@@ -73,6 +73,49 @@ run_once: true delegate_to: localhost +- name: Check if target directory exists for dirty check + stat: + path: "{{ bootlinux_9p_host_path }}" + register: git_dir_stat + run_once: true + delegate_to: localhost + when: + - not bootlinux_tree_set_by_cli|bool + +- name: Check if git tree is dirty + command: "git -C {{ bootlinux_9p_host_path }} status --porcelain" + register: git_status + changed_when: false + failed_when: false + run_once: true + delegate_to: localhost + when: + - not bootlinux_tree_set_by_cli|bool + - git_dir_stat.stat.exists + - git_dir_stat.stat.isdir + +- name: Fail if git tree has local modifications + fail: + msg: | + Local modifications exist in the destination: {{ bootlinux_9p_host_path }} + + The git tree is dirty with uncommitted changes. This prevents safe git operations. + + To resolve this, you can: + 1. Commit or stash your changes in {{ bootlinux_9p_host_path }} + 2. Reset the tree with: git -C {{ bootlinux_9p_host_path }} reset --hard + 3. Remove the directory and let kdevops clone fresh: rm -rf {{ bootlinux_9p_host_path }} + + Modified files: + {{ git_status.stdout }} + when: + - not bootlinux_tree_set_by_cli|bool + - git_dir_stat.stat.exists + - git_dir_stat.stat.isdir + - git_status.stdout | length > 0 + run_once: true + delegate_to: localhost + - name: git clone {{ target_linux_tree }} on the control node git: repo: "{{ target_linux_git }}"
playbooks/roles/bootlinux/tasks/build/builder.yml +33 -0
@@ -31,6 +31,39 @@ when: - ref_check.rc != 0 +- name: Check if target directory exists for dirty check + stat: + path: "{{ target_linux_dir_path }}" + register: git_dir_stat + +- name: Check if git tree is dirty + command: "git -C {{ target_linux_dir_path }} status --porcelain" + register: git_status + changed_when: false + failed_when: false + when: + - git_dir_stat.stat.exists + - git_dir_stat.stat.isdir + +- name: Fail if git tree has local modifications + fail: + msg: | + Local modifications exist in the destination: {{ target_linux_dir_path }} + + The git tree is dirty with uncommitted changes. This prevents safe git operations. + + To resolve this, you can: + 1. Commit or stash your changes in {{ target_linux_dir_path }} + 2. Reset the tree with: git -C {{ target_linux_dir_path }} reset --hard + 3. Remove the directory and let kdevops clone fresh: rm -rf {{ target_linux_dir_path }} + + Modified files: + {{ git_status.stdout }} + when: + - git_dir_stat.stat.exists + - git_dir_stat.stat.isdir + - git_status.stdout | length > 0 + - name: Clone {{ target_linux_tree }} ansible.builtin.git: repo: "{{ target_linux_git }}"
playbooks/roles/bootlinux/tasks/build/targets.yml +33 -0
@@ -32,6 +32,39 @@ when: - ref_check.rc != 0 +- name: Check if target directory exists for dirty check + stat: + path: "{{ target_linux_dir_path }}" + register: git_dir_stat + +- name: Check if git tree is dirty + command: "git -C {{ target_linux_dir_path }} status --porcelain" + register: git_status + changed_when: false + failed_when: false + when: + - git_dir_stat.stat.exists + - git_dir_stat.stat.isdir + +- name: Fail if git tree has local modifications + fail: + msg: | + Local modifications exist in the destination: {{ target_linux_dir_path }} + + The git tree is dirty with uncommitted changes. This prevents safe git operations. + + To resolve this, you can: + 1. Commit or stash your changes in {{ target_linux_dir_path }} + 2. Reset the tree with: git -C {{ target_linux_dir_path }} reset --hard + 3. Remove the directory and let kdevops clone fresh: rm -rf {{ target_linux_dir_path }} + + Modified files: + {{ git_status.stdout }} + when: + - git_dir_stat.stat.exists + - git_dir_stat.stat.isdir + - git_status.stdout | length > 0 + - name: git clone {{ target_linux_tree }} on the target nodes git: repo: "{{ target_linux_git }}"
Signature · 1 line
2.47.2
PATCH 5/8 bootlinux: add intelligent git repository detection and management
In reply to Luis ChamberlainAdd smart logic to infer when git clone is needed regardless of the bootlinux_tree_set_by_cli setting. This solves the issue where directories exist with files (like .config) but no .git repository, which previously caused the git clone to be skipped.
The new logic:
1. Checks if the target directory exists
2. Checks if .git directory exists within it
3. Infers git clone is needed if directory exists but .git doesn't
4. Clones the repository even when bootlinux_tree_set_by_cli is true
5. Ensures the correct ref is checked out if git exists but on wrong ref
This handles the common case where:
- Directory is pre-created for 9p mount support
- Configuration files are copied before git clone
- bootlinux_tree_set_by_cli persists from previous runs
The implementation also:
- Fetches updates if the target ref doesn't exist locally
- Switches to the correct ref if repository exists but is on wrong branch
- Maintains backward compatibility with existing workflows
- Applies consistently across all build methods (9p, targets, builder)
This makes the system more robust and user-friendly by intelligently handling partial setups and recovering from incomplete states.
--- playbooks/roles/bootlinux/tasks/build/9p.yml | 119 ++++++++++++++---- .../roles/bootlinux/tasks/build/builder.yml | 98 +++++++++++++-- .../roles/bootlinux/tasks/build/targets.yml | 98 +++++++++++++-- 3 files changed, 273 insertions(+), 42 deletions(-)
playbooks/roles/bootlinux/tasks/build/9p.yml +95 -24
@@ -26,23 +26,47 @@ run_once: true delegate_to: localhost -- name: Check if target directory exists when using 9p and Linux CLI was set +- name: Check if target directory exists stat: path: "{{ bootlinux_9p_host_path }}" register: target_directory_stat run_once: true delegate_to: localhost + +- name: Check if .git directory exists in target path + stat: + path: "{{ bootlinux_9p_host_path }}/.git" + register: git_directory_stat + run_once: true + delegate_to: localhost when: - - bootlinux_tree_set_by_cli|bool + - target_directory_stat.stat.exists -- name: Fail if target directory does not exist when using 9p and Linux CLI was set - fail: - msg: "The target directory {{ bootlinux_9p_host_path }} does not exist." +- name: Infer that git clone is needed when .git doesn't exist + set_fact: + needs_git_clone: true + when: + - target_directory_stat.stat.exists + - not git_directory_stat.stat.exists run_once: true delegate_to: localhost + +- name: Set needs_git_clone when directory doesn't exist + set_fact: + needs_git_clone: true when: - - bootlinux_tree_set_by_cli|bool - not target_directory_stat.stat.exists + run_once: true + delegate_to: localhost + +- name: Set needs_git_clone to false when .git exists + set_fact: + needs_git_clone: false + when: + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + run_once: true + delegate_to: localhost - name: Verify target git ref exists before cloning command: "git ls-remote {{ target_linux_git }} {{ active_linux_ref | default(target_linux_ref) }}"
@@ -50,7 +74,7 @@ run_once: true delegate_to: localhost when: - - not bootlinux_tree_set_by_cli|bool + - needs_git_clone|bool tags: [ 'clone'] - name: Fail if git ref does not exist
@@ -68,20 +92,11 @@ - If using A/B testing with different refs, ensure shallow cloning is disabled - The repository URL '{{ target_linux_git }}' is correct and accessible when: - - not bootlinux_tree_set_by_cli|bool + - needs_git_clone|bool - ref_check.rc != 0 run_once: true delegate_to: localhost -- name: Check if target directory exists for dirty check - stat: - path: "{{ bootlinux_9p_host_path }}" - register: git_dir_stat - run_once: true - delegate_to: localhost - when: - - not bootlinux_tree_set_by_cli|bool - - name: Check if git tree is dirty command: "git -C {{ bootlinux_9p_host_path }} status --porcelain" register: git_status
@@ -90,9 +105,9 @@ run_once: true delegate_to: localhost when: - - not bootlinux_tree_set_by_cli|bool - - git_dir_stat.stat.exists - - git_dir_stat.stat.isdir + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists - name: Fail if git tree has local modifications fail:
@@ -109,9 +124,9 @@ Modified files: {{ git_status.stdout }} when: - - not bootlinux_tree_set_by_cli|bool - - git_dir_stat.stat.exists - - git_dir_stat.stat.isdir + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists - git_status.stdout | length > 0 run_once: true delegate_to: localhost
@@ -129,9 +144,65 @@ until: not result.failed tags: [ 'clone'] when: - - not bootlinux_tree_set_by_cli|bool + - needs_git_clone|bool + run_once: true + delegate_to: localhost + +- name: Get current git ref when git exists but clone wasn't needed + command: "git -C {{ bootlinux_9p_host_path }} rev-parse HEAD" + register: current_ref + changed_when: false + run_once: true + delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + +- name: Get target ref SHA + command: "git -C {{ bootlinux_9p_host_path }} rev-parse {{ active_linux_ref | default(target_linux_ref) }}" + register: target_ref_sha + changed_when: false + failed_when: false run_once: true delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + +- name: Fetch updates if target ref doesn't exist locally + command: "git -C {{ bootlinux_9p_host_path }} fetch origin" + run_once: true + delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + +- name: Get target ref SHA after fetch + command: "git -C {{ bootlinux_9p_host_path }} rev-parse {{ active_linux_ref | default(target_linux_ref) }}" + register: target_ref_sha_after_fetch + changed_when: false + run_once: true + delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + +- name: Checkout target ref if not on correct ref + command: "git -C {{ bootlinux_9p_host_path }} checkout {{ active_linux_ref | default(target_linux_ref) }}" + run_once: true + delegate_to: localhost + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - (target_ref_sha.rc == 0 and current_ref.stdout != target_ref_sha.stdout) or + (target_ref_sha.rc != 0 and target_ref_sha_after_fetch is defined and target_ref_sha_after_fetch.rc == 0) - name: Copy kernel delta if requested on the control node template:
playbooks/roles/bootlinux/tasks/build/builder.yml +89 -9
133 lines not on this page · raw message
playbooks/roles/bootlinux/tasks/build/targets.yml +89 -9
134 lines not on this page · raw message
Signature · 1 line
2.47.2
PATCH 6/8 bootlinux: enhance A/B testing and repository management
In reply to Luis ChamberlainImprove git repository handling and A/B testing capabilities:
- Add intelligent fallback logic for git ref resolution
- Handle both direct refs and tag resolution gracefully
- Improve error handling for missing or invalid refs
- Add better debugging output for git operations
- Update defconfig to enable 4K reflink testing
- Exclude linux directory from newline checking script
These changes make the bootlinux workflow more robust when dealing with different git ref types and improve the A/B testing experience with better error reporting and fallback mechanisms.
--- defconfigs/xfs_reflink_lbs | 1 + playbooks/roles/bootlinux/tasks/build/9p.yml | 44 +++++++++++++++++-- .../roles/bootlinux/tasks/build/builder.yml | 39 ++++++++++++++-- .../roles/bootlinux/tasks/build/targets.yml | 39 ++++++++++++++-- scripts/ensure_newlines.py | 3 +- 5 files changed, 113 insertions(+), 13 deletions(-)
defconfigs/xfs_reflink_lbs +1 -0
@@ -20,6 +20,7 @@ CONFIG_FSTESTS_XFS_ENABLE_LBS=y CONFIG_FSTESTS_XFS_ENABLE_LBS_4KS=y CONFIG_FSTESTS_XFS_SECTION_REFLINK_ENABLED=y +CONFIG_FSTESTS_XFS_SECTION_REFLINK_4K=y CONFIG_FSTESTS_XFS_SECTION_REFLINK_8K_4KS=y CONFIG_FSTESTS_XFS_SECTION_REFLINK_16K_4KS=y CONFIG_FSTESTS_XFS_SECTION_REFLINK_32K_4KS=y
playbooks/roles/bootlinux/tasks/build/9p.yml +40 -4
72 lines not on this page · raw message
playbooks/roles/bootlinux/tasks/build/builder.yml +35 -4
@@ -137,24 +137,55 @@ - git_directory_stat.stat.exists - target_ref_sha.rc != 0 -- name: Get target ref SHA after fetch +- name: Try to resolve ref as direct ref after fetch command: "git -C {{ target_linux_dir_path }} rev-parse {{ target_linux_ref }}" - register: target_ref_sha_after_fetch + register: target_ref_sha_direct changed_when: false + failed_when: false when: - not needs_git_clone|bool - target_directory_stat.stat.exists - git_directory_stat.stat.exists - target_ref_sha.rc != 0 +- name: Try to resolve ref as remote branch if direct ref failed + command: "git -C {{ target_linux_dir_path }} rev-parse origin/{{ target_linux_ref }}" + register: target_ref_sha_remote + changed_when: false + failed_when: false + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + - target_ref_sha_direct.rc != 0 + +- name: Set resolved ref for checkout + set_fact: + resolved_ref: | + {%- if target_ref_sha.rc == 0 -%} + {{ target_linux_ref }} + {%- elif target_ref_sha_direct is defined and target_ref_sha_direct.rc == 0 -%} + {{ target_linux_ref }} + {%- elif target_ref_sha_remote is defined and target_ref_sha_remote.rc == 0 -%} + origin/{{ target_linux_ref }} + {%- else -%} + {{ target_linux_ref }} + {%- endif -%} + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - name: Checkout target ref if not on correct ref - command: "git -C {{ target_linux_dir_path }} checkout {{ target_linux_ref }}" + command: "git -C {{ target_linux_dir_path }} checkout {{ resolved_ref | default(target_linux_ref) }}" when: - not needs_git_clone|bool - target_directory_stat.stat.exists - git_directory_stat.stat.exists - (target_ref_sha.rc == 0 and current_ref.stdout != target_ref_sha.stdout) or - (target_ref_sha.rc != 0 and target_ref_sha_after_fetch is defined and target_ref_sha_after_fetch.rc == 0) + (target_ref_sha.rc != 0 and (target_ref_sha_direct is defined and target_ref_sha_direct.rc == 0)) or + (target_ref_sha.rc != 0 and (target_ref_sha_remote is defined and target_ref_sha_remote.rc == 0)) - name: Copy the kernel delta to the builder ansible.builtin.template:
playbooks/roles/bootlinux/tasks/build/targets.yml +35 -4
@@ -139,24 +139,55 @@ - git_directory_stat.stat.exists - target_ref_sha.rc != 0 -- name: Get target ref SHA after fetch +- name: Try to resolve ref as direct ref after fetch command: "git -C {{ target_linux_dir_path }} rev-parse {{ target_linux_ref }}" - register: target_ref_sha_after_fetch + register: target_ref_sha_direct changed_when: false + failed_when: false when: - not needs_git_clone|bool - target_directory_stat.stat.exists - git_directory_stat.stat.exists - target_ref_sha.rc != 0 +- name: Try to resolve ref as remote branch if direct ref failed + command: "git -C {{ target_linux_dir_path }} rev-parse origin/{{ target_linux_ref }}" + register: target_ref_sha_remote + changed_when: false + failed_when: false + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - target_ref_sha.rc != 0 + - target_ref_sha_direct.rc != 0 + +- name: Set resolved ref for checkout + set_fact: + resolved_ref: | + {%- if target_ref_sha.rc == 0 -%} + {{ target_linux_ref }} + {%- elif target_ref_sha_direct is defined and target_ref_sha_direct.rc == 0 -%} + {{ target_linux_ref }} + {%- elif target_ref_sha_remote is defined and target_ref_sha_remote.rc == 0 -%} + origin/{{ target_linux_ref }} + {%- else -%} + {{ target_linux_ref }} + {%- endif -%} + when: + - not needs_git_clone|bool + - target_directory_stat.stat.exists + - git_directory_stat.stat.exists + - name: Checkout target ref if not on correct ref - command: "git -C {{ target_linux_dir_path }} checkout {{ target_linux_ref }}" + command: "git -C {{ target_linux_dir_path }} checkout {{ resolved_ref | default(target_linux_ref) }}" when: - not needs_git_clone|bool - target_directory_stat.stat.exists - git_directory_stat.stat.exists - (target_ref_sha.rc == 0 and current_ref.stdout != target_ref_sha.stdout) or - (target_ref_sha.rc != 0 and target_ref_sha_after_fetch is defined and target_ref_sha_after_fetch.rc == 0) + (target_ref_sha.rc != 0 and (target_ref_sha_direct is defined and target_ref_sha_direct.rc == 0)) or + (target_ref_sha.rc != 0 and (target_ref_sha_remote is defined and target_ref_sha_remote.rc == 0)) - name: Copy kernel delta if requested on the target nodes template:
scripts/ensure_newlines.py +2 -1
@@ -45,7 +45,8 @@ def main(): dirs[:] = [ d for d in dirs - if not d.startswith(".") and d not in ["__pycache__", "node_modules"] + if not d.startswith(".") + and d not in ["__pycache__", "node_modules", "linux"] ] for file in files:
Signature · 1 line
2.47.2
PATCH 7/8 fstests: add make target for running tests on all hosts
In reply to Luis ChamberlainAdd a new 'fstests-tests' make target that runs tests on both baseline and dev hosts simultaneously. This target is useful for A/B testing scenarios where you want to run the same tests on multiple host configurations in parallel.
The target uses the same FSTESTS_DYNAMIC_RUNTIME_VARS as the existing baseline and dev targets but limits execution to both baseline and dev host groups together.
--- workflows/fstests/Makefile | 10 ++++++++++ 1 file changed, 10 insertions(+)
workflows/fstests/Makefile +10 -0
@@ -198,6 +198,15 @@ fstests-dev: $(KDEVOPS_EXTRA_VARS) '{ $(FSTESTS_DYNAMIC_RUNTIME_VARS) }' \ --extra-vars=@./extra_vars.yaml $(LIMIT_HOSTS) +fstests-tests: $(KDEVOPS_EXTRA_VARS) + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ + --limit 'baseline:dev' \ + playbooks/fstests.yml \ + --tags vars,run_tests,copy_results \ + --extra-vars \ + '{ $(FSTESTS_DYNAMIC_RUNTIME_VARS) }' \ + --extra-vars=@./extra_vars.yaml $(LIMIT_HOSTS) + fstests-baseline-results-tfb-ls: $(KDEVOPS_EXTRA_VARS) $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ --limit 'baseline' \
@@ -249,6 +258,7 @@ fstests-help-menu: @echo "fstests-kdevops-setup - Install kdevops specific files, the fstests and running test targets also runs this" @echo "fstests-baseline - Run fstests on baseline hosts and collect results" @echo "fstests-dev - Run fstests on dev hosts and collect results" + @echo "fstests-tests - Run fstests on both baseline and dev hosts simultaneously" @echo "" @echo "fstests-config - Generates the filesystem configuration file only onto target systems" @echo "fstests-config-debug - Generates the filesystem configuration file locally, useful for debugging"
Signature · 1 line
2.47.2
PATCH 8/8 monitoring: integrate monitoring collection into fstests workflow
In reply to Luis ChamberlainFix monitoring data collection to work properly during fstests runs by changing from include_role to import_tasks. The include_role directive with tasks_from parameter wasn't executing properly due to tag filtering during playbook execution, preventing monitoring data from being collected.
Using import_tasks ensures the monitoring tasks are statically included at parse time and properly executed when the appropriate tags are present. This allows monitoring to run on all hosts and collect folio migration statistics during test execution.
The monitoring documentation has also been updated to reflect that it's now a shared service available to all workflows.
--- Kconfig | 4 ++++ README.md | 1 + playbooks/roles/fstests/tasks/main.yml | 14 ++++++++++++++ 3 files changed, 19 insertions(+)
Kconfig +4 -0
@@ -79,6 +79,10 @@ menu "Target workflows" source "kconfigs/workflows/Kconfig" endmenu +menu "Monitors" +source "kconfigs/monitors/Kconfig" +endmenu + menu "Kdevops configuration" source "kconfigs/Kconfig.kdevops" endmenu
README.md +1 -0
@@ -306,6 +306,7 @@ Below is kdevops' recommended documentation reading. * [kdevops' evolving make help](docs/evolving-make-help.md) * [kdevops configuration](docs/kdevops-configuration.md) * [kdevops mirror support](docs/kdevops-mirror.md) + * [kdevops monitoring services](docs/monitoring.md) * [kdevops first run](docs/kdevops-first-run.md) * [kdevops running make](docs/running-make.md) * [kdevops libvirt storage pool considerations](docs/libvirt-storage-pool.md)
playbooks/roles/fstests/tasks/main.yml +14 -0
@@ -1235,6 +1235,13 @@ when: - fstests_skip_run|bool +# Start monitoring services before running tests +- import_tasks: ../../monitoring/tasks/monitor_run.yml + when: + - kdevops_run_fstests|bool + - enable_monitoring|default(false)|bool + tags: [ 'oscheck', 'fstests', 'run_tests', 'monitoring', 'monitor_run' ] + # Recent environments runs are showing that environment variables # set below are not propagated. So best to stuff what you need # into the .kdevops_fstests_setup file which is sourced by root.
@@ -1281,6 +1288,13 @@ when: - kdevops_run_fstests|bool +# Stop monitoring services and collect data after running tests +- import_tasks: ../../monitoring/tasks/monitor_collect.yml + when: + - kdevops_run_fstests|bool + - enable_monitoring|default(false)|bool + tags: [ 'oscheck', 'fstests', 'run_tests', 'monitoring', 'monitor_collect' ] + - name: Remove watchdog hint that tests have started local_action: file path="{{ fstests_workflow_dir }}/.begin" state=absent tags: [ 'oscheck', 'fstests', 'run_tests' ]
Signature · 1 line
2.47.2
Re: [PATCH 8/8] monitoring: integrate monitoring collection into fstests workflow
Quoted 10 lines · Luis Chamberlain
Fix monitoring data collection to work properly during fstests runs by changing from
include_roletoimport_tasks. Theinclude_roledirective withtasks_fromparameter wasn't executing properly due to tag filtering during playbook execution, preventing monitoring data from being collected.Using
import_tasksensures the monitoring tasks are statically included at parse time and properly executed when the appropriate tags are present. This allows monitoring to run on all hosts and collect folio migration statistics during test execution.
Quoted 5 lines
The monitoring documentation has also been updated to reflect that it's now a shared service available to all workflows.
Generated-by Claude AISigned-off-by Luis Chamberlain <[email protected]>
Bah, this patch can be ignored for now, I forgot to git am a few things.
Luis
Re: [PATCH 8/8] monitoring: integrate monitoring collection into fstests workflow
Quoted 11 lines · Luis Chamberlain
Quoted 10 lines · Luis Chamberlain
Fix monitoring data collection to work properly during fstests runs by changing from
include_roletoimport_tasks. Theinclude_roledirective withtasks_fromparameter wasn't executing properly due to tag filtering during playbook execution, preventing monitoring data from being collected.Using
import_tasksensures the monitoring tasks are statically included at parse time and properly executed when the appropriate tags are present. This allows monitoring to run on all hosts and collect folio migration statistics during test execution.
Quoted 7 lines
Quoted 5 lines
The monitoring documentation has also been updated to reflect that it's now a shared service available to all workflows.
Generated-by Claude AISigned-off-by Luis Chamberlain <[email protected]>Bah, this patch can be ignored for now, I forgot to git am a few things.
OK here's a v2 of that patch follows and a branch with *all* my pending changes:
https://github.com/linux-kdevops/kdevops/tree/20250811-monitor-v2
From 75bbc91c5cdaced1ab294b979cf58c13d8ead34c Mon Sep 17 00:00:00 2001
From: Luis Chamberlain <[email protected]>
Date: Mon, 11 Aug 2025 15:33:17 -0700
Subject: [PATCH v2] monitoring: add monitoring framework for workflow execution
Add a flexible monitoring framework that collects system metrics during workflow execution. The framework supports background monitoring services that automatically start before workflows and collect results afterward.
Initial implementation includes:
- Core monitoring infrastructure with Kconfig integration
- Folio migration statistics monitor (for developmental kernel features)
- Integration with fstests workflow
- Result collection and visualization support
- Documentation for adding new monitors and integrating with workflows
The monitoring system is designed to be modular, allowing workflows to opt-in and new monitors to be easily added. Results are stored in workflow-specific directories and can include both raw data and visualizations.
--- Kconfig | 4 + README.md | 1 + docs/monitoring.md | 279 ++++++++++++++++++ kconfigs/monitors/Kconfig | 74 +++++ playbooks/roles/fstests/tasks/main.yml | 14 + playbooks/roles/monitoring/defaults/main.yml | 18 ++ .../monitoring/files/plot_migration_stats.py | 220 ++++++++++++++ playbooks/roles/monitoring/tasks/main.yml | 23 ++ .../monitoring/tasks/monitor_collect.yml | 205 +++++++++++++ .../roles/monitoring/tasks/monitor_run.yml | 83 ++++++ 10 files changed, 921 insertions(+) create mode 100644 docs/monitoring.md create mode 100644 kconfigs/monitors/Kconfig create mode 100644 playbooks/roles/monitoring/defaults/main.yml create mode 100755 playbooks/roles/monitoring/files/plot_migration_stats.py create mode 100644 playbooks/roles/monitoring/tasks/main.yml create mode 100644 playbooks/roles/monitoring/tasks/monitor_collect.yml create mode 100644 playbooks/roles/monitoring/tasks/monitor_run.yml
Kconfig +4 -0
@@ -79,6 +79,10 @@ menu "Target workflows" source "kconfigs/workflows/Kconfig" endmenu +menu "Monitors" +source "kconfigs/monitors/Kconfig" +endmenu + menu "Kdevops configuration" source "kconfigs/Kconfig.kdevops" endmenu
README.md +1 -0
@@ -306,6 +306,7 @@ Below is kdevops' recommended documentation reading. * [kdevops' evolving make help](docs/evolving-make-help.md) * [kdevops configuration](docs/kdevops-configuration.md) * [kdevops mirror support](docs/kdevops-mirror.md) + * [kdevops monitoring services](docs/monitoring.md) * [kdevops first run](docs/kdevops-first-run.md) * [kdevops running make](docs/running-make.md) * [kdevops libvirt storage pool considerations](docs/libvirt-storage-pool.md)
docs/monitoring.md new file +279 -0
279 lines not on this page · raw message
kconfigs/monitors/Kconfig new file +74 -0
74 lines not on this page · raw message
playbooks/roles/fstests/tasks/main.yml +14 -0
@@ -1235,6 +1235,13 @@ when: - fstests_skip_run|bool +# Start monitoring services before running tests +- import_tasks: ../../monitoring/tasks/monitor_run.yml + when: + - kdevops_run_fstests|bool + - enable_monitoring|default(false)|bool + tags: [ 'oscheck', 'fstests', 'run_tests', 'monitoring', 'monitor_run' ] + # Recent environments runs are showing that environment variables # set below are not propagated. So best to stuff what you need # into the .kdevops_fstests_setup file which is sourced by root.
@@ -1281,6 +1288,13 @@ when: - kdevops_run_fstests|bool +# Stop monitoring services and collect data after running tests +- import_tasks: ../../monitoring/tasks/monitor_collect.yml + when: + - kdevops_run_fstests|bool + - enable_monitoring|default(false)|bool + tags: [ 'oscheck', 'fstests', 'run_tests', 'monitoring', 'monitor_collect' ] + - name: Remove watchdog hint that tests have started local_action: file path="{{ fstests_workflow_dir }}/.begin" state=absent tags: [ 'oscheck', 'fstests', 'run_tests' ]
playbooks/roles/monitoring/defaults/main.yml new file +18 -0
@@ -0,0 +1,18 @@ +--- +# Default values for monitoring role + +# Enable monitoring services +enable_monitoring: false + +# Enable developmental statistics +monitor_developmental_stats: false + +# Enable folio migration monitoring +monitor_folio_migration: false + +# Folio migration monitoring interval in seconds +monitor_folio_migration_interval: 60 + +# Base path to store monitoring results on the control host +# Workflows can override by setting monitoring_results_base_path +monitoring_results_base_path: "{{ topdir_path }}/workflows/fstests/results/monitoring"
playbooks/roles/monitoring/files/plot_migration_stats.py new file +220 -0
220 lines not on this page · raw message
playbooks/roles/monitoring/tasks/main.yml new file +23 -0
@@ -0,0 +1,23 @@ +--- +- name: Import optional extra_args file + include_vars: "{{ item }}" + ignore_errors: yes + with_first_found: + - files: + - "../extra_vars.yml" + - "../extra_vars.yaml" + - "../extra_vars.json" + skip: true + tags: vars + +- name: Include monitor_run tasks + include_tasks: monitor_run.yml + when: + - enable_monitoring|default(false)|bool + tags: [ 'monitoring', 'monitor_run' ] + +- name: Include monitor_collect tasks + include_tasks: monitor_collect.yml + when: + - enable_monitoring|default(false)|bool + tags: [ 'monitoring', 'monitor_collect' ]
playbooks/roles/monitoring/tasks/monitor_collect.yml new file +205 -0
205 lines not on this page · raw message
playbooks/roles/monitoring/tasks/monitor_run.yml new file +83 -0
83 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [PATCH 8/8] monitoring: integrate monitoring collection into fstests workflow
Quoted 18 lines · Luis Chamberlain
Quoted 17 lines · Luis Chamberlain
Quoted 15 lines · Luis Chamberlain
Fix monitoring data collection to work properly during fstests runs by changing from
include_roletoimport_tasks. Theinclude_roledirective withtasks_fromparameter wasn't executing properly due to tag filtering during playbook execution, preventing monitoring data from being collected.Using
import_tasksensures the monitoring tasks are statically included at parse time and properly executed when the appropriate tags are present. This allows monitoring to run on all hosts and collect folio migration statistics during test execution.The monitoring documentation has also been updated to reflect that it's now a shared service available to all workflows.
Generated-by Claude AISigned-off-by Luis Chamberlain <[email protected]>
Quoted 6 lines
Quoted 1 line
Bah, this patch can be ignored for now, I forgot to git am a few things.
OK here's a v2 of that patch follows and a branch with *all* my pending changes:
https://github.com/linux-kdevops/kdevops/tree/20250811-monitor-v2
I've pushed a better version of this, along with some fstests watchdog enhancements.
Luis