Skip to contentSkip to conversation

linux-ab enhancements + monitor support

PATCH In review Luis Chamberlain 1 participant 12 messages Aug 11, 2025 – Aug 14, 2025

lore
Share & tools
lore
#0 lore raw

cover linux-ab enhancements + monitor support

These are a collection of practical enhancements developed as I tested Linux AB testing with parallel writeback developmental patches to help do analysis on impact on memory management. The last patch adds the ability to do experimental (non upstream) monitoring of different types. We could obviously later add monitoring for existing upstream knobs but in this case we want to track folio migration success rates and so are using a currently out of tree knob.

Luis Chamberlain (8):
bootlinux: use different kernel for A/B testing by default
bootlinux: add support for custom refs on dev kernels on the CLI
bootlinux: add git ref verification before cloning
bootlinux: add git dirty check before cloning
bootlinux: add intelligent git repository detection and management
bootlinux: enhance A/B testing and repository management
fstests: add make target for running tests on all hosts
monitoring: integrate monitoring collection into fstests workflow

 Kconfig                                       |   4 +
 README.md                                     |   1 +
 defconfigs/xfs_reflink_lbs                    |   1 +
 kconfigs/Kconfig.kdevops                      |   8 +-
 playbooks/roles/bootlinux/tasks/build/9p.yml  | 193 +++++++++++++++++-
 .../roles/bootlinux/tasks/build/builder.yml   | 165 +++++++++++++++
 .../roles/bootlinux/tasks/build/targets.yml   | 166 +++++++++++++++
 playbooks/roles/fstests/tasks/main.yml        |  14 ++
 scripts/ensure_newlines.py                    |   3 +-
 workflows/fstests/Makefile                    |  10 +
 workflows/linux/Kconfig                       |  18 +-
 11 files changed, 571 insertions(+), 12 deletions(-)
Signature · 1 line
2.47.2
#1 lore raw

PATCH 1/8 bootlinux: use different kernel for A/B testing by default

If you are doing A/B testing then by default you'd want to test different kernels typically. So enable that by default. This will help make our CIs easier to write leveraging defconfigs with less crap in them.

---
 workflows/linux/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
workflows/linux/Kconfig +1 -1
index 44456904..aa7c6ec8 100644
@@ -385,7 +385,7 @@ if KDEVOPS_BASELINE_AND_DEV
 
 choice
 	prompt "A/B kernel testing configuration"
-	default BOOTLINUX_AB_SAME_REF
+	default BOOTLINUX_AB_DIFFERENT_REF
 	help
 	  When A/B testing is enabled, you can choose to use the same
 	  kernel reference for both baseline and dev nodes, or specify
Signature · 1 line
2.47.2
#2 lore raw

PATCH 2/8 bootlinux: add support for custom refs on dev kernels on the CLI

In reply to Luis Chamberlain

We already have support to easily test arbitrary kernels and refs by just using the LINUX_TREE and LINUX_REF environment variables. We leverage this for our CIs:

* LINUX_TREE: the target tree for baseline group (A)
* LINUX_REF: the target tree ref the baseline group (A)

Now that we have AB-testing support we want to first add support so that a simple TEST_AB=y will enable KDEVOPS_BASELINE_AND_DEV and then we want to be able to customize the target dev tree and ref. We do this with two other environment variables:

* LINUX_DEV_TREE: the target tree for dev group (B)
* LINUX_DEV_REF: the target tree ref the dev group (B)

You just need to make sure to also pass TEST_AB=y.

By leveraging this we can easily use AB testing in the future on CIs to compare and contrast different kernel releases against any target workload we have.

---
 kconfigs/Kconfig.kdevops |  8 +++++++-
 workflows/linux/Kconfig  | 16 ++++++++++++++--
 2 files changed, 21 insertions(+), 3 deletions(-)
kconfigs/Kconfig.kdevops +7 -1
index c2362adf..431d579a 100644
@@ -51,6 +51,11 @@ config KDEVOPS_HOSTS_PREFIX_SET_BY_CLI
 	bool
 	default $(shell, scripts/check-cli-set-var.sh KDEVOPS_HOSTS_PREFIX)
 
+config KDEVOPS_BASELINE_AND_DEV_SET_BY_CLI
+	bool
+	output yaml
+	default $(shell, scripts/check-cli-set-var.sh TEST_AB)
+
 config KDEVOPS_HOSTS_PREFIX
 	string "The hostname prefix to use for nodes"
 	output yaml
@@ -78,7 +83,8 @@ config KDEVOPS_CUSTOM_SSH_KEXALGORITHMS
 
 config KDEVOPS_BASELINE_AND_DEV
 	bool "Enable both a baseline and development system per target test"
-	default n
+	default n if !KDEVOPS_BASELINE_AND_DEV_SET_BY_CLI
+	default y if KDEVOPS_BASELINE_AND_DEV_SET_BY_CLI
 	help
 	  By default kdevops will only spawn a baseline target node (local
 	  virtualization or cloud node) for your Linux kernel testing and
workflows/linux/Kconfig +14 -2
index aa7c6ec8..2469ef97 100644
@@ -10,6 +10,16 @@ config BOOTLINUX_TREE_REF_SET_BY_CLI
 	output yaml
 	default $(shell, scripts/check-cli-set-var.sh LINUX_TREE_REF)
 
+config BOOTLINUX_DEV_TREE_SET_BY_CLI
+	bool
+	output yaml
+	default $(shell, scripts/check-cli-set-var.sh LINUX_DEV_TREE)
+
+config BOOTLINUX_DEV_TREE_REF_SET_BY_CLI
+	bool
+	output yaml
+	default $(shell, scripts/check-cli-set-var.sh LINUX_DEV_TREE_REF)
+
 config BOOTLINUX_HAS_PURE_IOMAP_CONFIG
 	bool
 
@@ -415,7 +425,8 @@ if BOOTLINUX_AB_DIFFERENT_REF
 config BOOTLINUX_DEV_TREE
 	string "Development kernel tree URL"
 	output yaml
-	default BOOTLINUX_TREE
+	default BOOTLINUX_TREE if !BOOTLINUX_DEV_TREE_SET_BY_CLI
+	default $(shell, ./scripts/append-makefile-vars.sh $(LINUX_DEV_TREE)) if BOOTLINUX_DEV_TREE_SET_BY_CLI
 	help
 	  Git tree URL for the development kernel. If left empty or same
 	  as the baseline tree, the same tree will be used with a different
@@ -424,7 +435,8 @@ config BOOTLINUX_DEV_TREE
 config TARGET_LINUX_DEV_REF
 	string "Development kernel reference"
 	output yaml
-	default $(shell, scripts/infer_last_stable_kernel.sh)
+	default $(shell, scripts/infer_last_stable_kernel.sh) if !BOOTLINUX_DEV_TREE_REF_SET_BY_CLI
+	default $(shell, ./scripts/append-makefile-vars.sh $(LINUX_DEV_TREE_REF)) if BOOTLINUX_DEV_TREE_REF_SET_BY_CLI
 	help
 	  Git reference (branch, tag, or commit) for the development kernel.
 	  This should be different from the baseline reference to enable
Signature · 1 line
2.47.2
#3 lore raw

PATCH 3/8 bootlinux: add git ref verification before cloning

In reply to Luis Chamberlain

Add preliminary verification tasks to check if the target git ref exists before attempting to clone the Linux kernel repository. This prevents confusing git clone failures and provides clear, actionable error messages to users.

The verification is particularly important for A/B testing scenarios where:
- Different kernel refs may be used for baseline vs development builds
- Shallow clones might not contain all required refs
- Users may specify refs that don't exist in the repository

Each build method now verifies the ref availability:
- 9p.yml: Verifies active_linux_ref (or target_linux_ref fallback)
- targets.yml: Verifies target_linux_ref on target nodes
- builder.yml: Verifies target_linux_ref on builder nodes

The error messages guide users to:
1. Check if the ref actually exists in the repository
2. Disable shallow cloning when using A/B testing with different refs
3. Verify the repository URL is correct and accessible

This change improves the user experience by failing fast with helpful diagnostics rather than letting git clone fail with cryptic errors.

Generated-by Claude AI
---
 playbooks/roles/bootlinux/tasks/build/9p.yml  | 29 +++++++++++++++++++
 .../roles/bootlinux/tasks/build/builder.yml   | 21 ++++++++++++++
 .../roles/bootlinux/tasks/build/targets.yml   | 22 ++++++++++++++
 3 files changed, 72 insertions(+)
playbooks/roles/bootlinux/tasks/build/9p.yml +29 -0
index 1951e50e..98cbcb3c 100644
@@ -44,6 +44,35 @@
     - bootlinux_tree_set_by_cli|bool
     - not target_directory_stat.stat.exists
 
+- name: Verify target git ref exists before cloning
+  command: "git ls-remote {{ target_linux_git }} {{ active_linux_ref | default(target_linux_ref) }}"
+  register: ref_check
+  run_once: true
+  delegate_to: localhost
+  when:
+    - not bootlinux_tree_set_by_cli|bool
+  tags: [ 'clone']
+
+- name: Fail if git ref does not exist
+  fail:
+    msg: |
+      Failed to verify git ref '{{ active_linux_ref | default(target_linux_ref) }}' exists in repository '{{ target_linux_git }}'.
+
+      This typically happens when:
+      1. The ref (branch/tag/commit) doesn't exist in the repository
+      2. You're using A/B testing with a shallow clone that doesn't contain the required ref
+      3. The repository URL is incorrect or inaccessible
+
+      Please verify:
+      - The ref '{{ active_linux_ref | default(target_linux_ref) }}' exists in the repository
+      - If using A/B testing with different refs, ensure shallow cloning is disabled
+      - The repository URL '{{ target_linux_git }}' is correct and accessible
+  when:
+    - not bootlinux_tree_set_by_cli|bool
+    - ref_check.rc != 0
+  run_once: true
+  delegate_to: localhost
+
 - name: git clone {{ target_linux_tree }} on the control node
   git:
     repo: "{{ target_linux_git }}"
playbooks/roles/bootlinux/tasks/build/builder.yml +21 -0
index c4c4b950..73cc6694 100644
@@ -10,6 +10,27 @@
     - target_linux_install_b4
     - ansible_os_family == "Debian"
 
+- name: Verify target git ref exists before cloning
+  command: "git ls-remote {{ target_linux_git }} {{ target_linux_ref }}"
+  register: ref_check
+
+- name: Fail if git ref does not exist
+  fail:
+    msg: |
+      Failed to verify git ref '{{ target_linux_ref }}' exists in repository '{{ target_linux_git }}'.
+
+      This typically happens when:
+      1. The ref (branch/tag/commit) doesn't exist in the repository
+      2. You're using A/B testing with a shallow clone that doesn't contain the required ref
+      3. The repository URL is incorrect or inaccessible
+
+      Please verify:
+      - The ref '{{ target_linux_ref }}' exists in the repository
+      - If using A/B testing with different refs, ensure shallow cloning is disabled
+      - The repository URL '{{ target_linux_git }}' is correct and accessible
+  when:
+    - ref_check.rc != 0
+
 - name: Clone {{ target_linux_tree }}
   ansible.builtin.git:
     repo: "{{ target_linux_git }}"
playbooks/roles/bootlinux/tasks/build/targets.yml +22 -0
index 36339876..81465cc6 100644
@@ -10,6 +10,28 @@
     - target_linux_install_b4
     - ansible_facts['os_family']|lower != 'debian'
 
+- name: Verify target git ref exists before cloning
+  command: "git ls-remote {{ target_linux_git }} {{ target_linux_ref }}"
+  register: ref_check
+  tags: [ 'clone']
+
+- name: Fail if git ref does not exist
+  fail:
+    msg: |
+      Failed to verify git ref '{{ target_linux_ref }}' exists in repository '{{ target_linux_git }}'.
+
+      This typically happens when:
+      1. The ref (branch/tag/commit) doesn't exist in the repository
+      2. You're using A/B testing with a shallow clone that doesn't contain the required ref
+      3. The repository URL is incorrect or inaccessible
+
+      Please verify:
+      - The ref '{{ target_linux_ref }}' exists in the repository
+      - If using A/B testing with different refs, ensure shallow cloning is disabled
+      - The repository URL '{{ target_linux_git }}' is correct and accessible
+  when:
+    - ref_check.rc != 0
+
 - name: git clone {{ target_linux_tree }} on the target nodes
   git:
     repo: "{{ target_linux_git }}"
Signature · 1 line
2.47.2
#4 lore raw

PATCH 4/8 bootlinux: add git dirty check before cloning

In reply to Luis Chamberlain

Add preliminary checks to detect if the git tree has local modifications before attempting to clone or update. This provides clearer error messages when git operations fail due to uncommitted changes.

The issue was discovered when prior kdevops commits inadvertently modified files in the linux kernel tree through overly broad make style operations. When git clone/update encounters a dirty tree, it fails with a generic "Local modifications exist" error that doesn't guide users on resolution.

Each build method now checks for dirty trees:
- 9p.yml: Checks bootlinux_9p_host_path on control node
- targets.yml: Checks target_linux_dir_path on target nodes
- builder.yml: Checks target_linux_dir_path on builder nodes

The error messages provide clear resolution steps:
1. Commit or stash changes
2. Hard reset the tree
3. Remove the directory for a fresh clone

This change improves debugging by:
- Failing fast with actionable error messages
- Showing which files are modified
- Providing multiple resolution options
- Preventing confusing git errors downstream

Generated-by Claude AI
---
 playbooks/roles/bootlinux/tasks/build/9p.yml  | 43 +++++++++++++++++++
 .../roles/bootlinux/tasks/build/builder.yml   | 33 ++++++++++++++
 .../roles/bootlinux/tasks/build/targets.yml   | 33 ++++++++++++++
 3 files changed, 109 insertions(+)
playbooks/roles/bootlinux/tasks/build/9p.yml +43 -0
index 98cbcb3c..5c9489c3 100644
@@ -73,6 +73,49 @@
   run_once: true
   delegate_to: localhost
 
+- name: Check if target directory exists for dirty check
+  stat:
+    path: "{{ bootlinux_9p_host_path }}"
+  register: git_dir_stat
+  run_once: true
+  delegate_to: localhost
+  when:
+    - not bootlinux_tree_set_by_cli|bool
+
+- name: Check if git tree is dirty
+  command: "git -C {{ bootlinux_9p_host_path }} status --porcelain"
+  register: git_status
+  changed_when: false
+  failed_when: false
+  run_once: true
+  delegate_to: localhost
+  when:
+    - not bootlinux_tree_set_by_cli|bool
+    - git_dir_stat.stat.exists
+    - git_dir_stat.stat.isdir
+
+- name: Fail if git tree has local modifications
+  fail:
+    msg: |
+      Local modifications exist in the destination: {{ bootlinux_9p_host_path }}
+
+      The git tree is dirty with uncommitted changes. This prevents safe git operations.
+
+      To resolve this, you can:
+      1. Commit or stash your changes in {{ bootlinux_9p_host_path }}
+      2. Reset the tree with: git -C {{ bootlinux_9p_host_path }} reset --hard
+      3. Remove the directory and let kdevops clone fresh: rm -rf {{ bootlinux_9p_host_path }}
+
+      Modified files:
+      {{ git_status.stdout }}
+  when:
+    - not bootlinux_tree_set_by_cli|bool
+    - git_dir_stat.stat.exists
+    - git_dir_stat.stat.isdir
+    - git_status.stdout | length > 0
+  run_once: true
+  delegate_to: localhost
+
 - name: git clone {{ target_linux_tree }} on the control node
   git:
     repo: "{{ target_linux_git }}"
playbooks/roles/bootlinux/tasks/build/builder.yml +33 -0
index 73cc6694..d36815ed 100644
@@ -31,6 +31,39 @@
   when:
     - ref_check.rc != 0
 
+- name: Check if target directory exists for dirty check
+  stat:
+    path: "{{ target_linux_dir_path }}"
+  register: git_dir_stat
+
+- name: Check if git tree is dirty
+  command: "git -C {{ target_linux_dir_path }} status --porcelain"
+  register: git_status
+  changed_when: false
+  failed_when: false
+  when:
+    - git_dir_stat.stat.exists
+    - git_dir_stat.stat.isdir
+
+- name: Fail if git tree has local modifications
+  fail:
+    msg: |
+      Local modifications exist in the destination: {{ target_linux_dir_path }}
+
+      The git tree is dirty with uncommitted changes. This prevents safe git operations.
+
+      To resolve this, you can:
+      1. Commit or stash your changes in {{ target_linux_dir_path }}
+      2. Reset the tree with: git -C {{ target_linux_dir_path }} reset --hard
+      3. Remove the directory and let kdevops clone fresh: rm -rf {{ target_linux_dir_path }}
+
+      Modified files:
+      {{ git_status.stdout }}
+  when:
+    - git_dir_stat.stat.exists
+    - git_dir_stat.stat.isdir
+    - git_status.stdout | length > 0
+
 - name: Clone {{ target_linux_tree }}
   ansible.builtin.git:
     repo: "{{ target_linux_git }}"
playbooks/roles/bootlinux/tasks/build/targets.yml +33 -0
index 81465cc6..414602ab 100644
@@ -32,6 +32,39 @@
   when:
     - ref_check.rc != 0
 
+- name: Check if target directory exists for dirty check
+  stat:
+    path: "{{ target_linux_dir_path }}"
+  register: git_dir_stat
+
+- name: Check if git tree is dirty
+  command: "git -C {{ target_linux_dir_path }} status --porcelain"
+  register: git_status
+  changed_when: false
+  failed_when: false
+  when:
+    - git_dir_stat.stat.exists
+    - git_dir_stat.stat.isdir
+
+- name: Fail if git tree has local modifications
+  fail:
+    msg: |
+      Local modifications exist in the destination: {{ target_linux_dir_path }}
+
+      The git tree is dirty with uncommitted changes. This prevents safe git operations.
+
+      To resolve this, you can:
+      1. Commit or stash your changes in {{ target_linux_dir_path }}
+      2. Reset the tree with: git -C {{ target_linux_dir_path }} reset --hard
+      3. Remove the directory and let kdevops clone fresh: rm -rf {{ target_linux_dir_path }}
+
+      Modified files:
+      {{ git_status.stdout }}
+  when:
+    - git_dir_stat.stat.exists
+    - git_dir_stat.stat.isdir
+    - git_status.stdout | length > 0
+
 - name: git clone {{ target_linux_tree }} on the target nodes
   git:
     repo: "{{ target_linux_git }}"
Signature · 1 line
2.47.2
#5 lore raw

PATCH 5/8 bootlinux: add intelligent git repository detection and management

In reply to Luis Chamberlain

Add smart logic to infer when git clone is needed regardless of the bootlinux_tree_set_by_cli setting. This solves the issue where directories exist with files (like .config) but no .git repository, which previously caused the git clone to be skipped.

The new logic:
1. Checks if the target directory exists
2. Checks if .git directory exists within it
3. Infers git clone is needed if directory exists but .git doesn't
4. Clones the repository even when bootlinux_tree_set_by_cli is true
5. Ensures the correct ref is checked out if git exists but on wrong ref

This handles the common case where:
- Directory is pre-created for 9p mount support
- Configuration files are copied before git clone
- bootlinux_tree_set_by_cli persists from previous runs

The implementation also:
- Fetches updates if the target ref doesn't exist locally
- Switches to the correct ref if repository exists but is on wrong branch
- Maintains backward compatibility with existing workflows
- Applies consistently across all build methods (9p, targets, builder)

This makes the system more robust and user-friendly by intelligently handling partial setups and recovering from incomplete states.

Generated-by Claude AI
---
 playbooks/roles/bootlinux/tasks/build/9p.yml  | 119 ++++++++++++++----
 .../roles/bootlinux/tasks/build/builder.yml   |  98 +++++++++++++--
 .../roles/bootlinux/tasks/build/targets.yml   |  98 +++++++++++++--
 3 files changed, 273 insertions(+), 42 deletions(-)
playbooks/roles/bootlinux/tasks/build/9p.yml +95 -24
index 5c9489c3..d0ae61ad 100644
@@ -26,23 +26,47 @@
   run_once: true
   delegate_to: localhost
 
-- name: Check if target directory exists when using 9p and Linux CLI was set
+- name: Check if target directory exists
   stat:
     path: "{{ bootlinux_9p_host_path }}"
   register: target_directory_stat
   run_once: true
   delegate_to: localhost
+
+- name: Check if .git directory exists in target path
+  stat:
+    path: "{{ bootlinux_9p_host_path }}/.git"
+  register: git_directory_stat
+  run_once: true
+  delegate_to: localhost
   when:
-    - bootlinux_tree_set_by_cli|bool
+    - target_directory_stat.stat.exists
 
-- name: Fail if target directory does not exist when using 9p and Linux CLI was set
-  fail:
-    msg: "The target directory {{ bootlinux_9p_host_path }} does not exist."
+- name: Infer that git clone is needed when .git doesn't exist
+  set_fact:
+    needs_git_clone: true
+  when:
+    - target_directory_stat.stat.exists
+    - not git_directory_stat.stat.exists
   run_once: true
   delegate_to: localhost
+
+- name: Set needs_git_clone when directory doesn't exist
+  set_fact:
+    needs_git_clone: true
   when:
-    - bootlinux_tree_set_by_cli|bool
     - not target_directory_stat.stat.exists
+  run_once: true
+  delegate_to: localhost
+
+- name: Set needs_git_clone to false when .git exists
+  set_fact:
+    needs_git_clone: false
+  when:
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+  run_once: true
+  delegate_to: localhost
 
 - name: Verify target git ref exists before cloning
   command: "git ls-remote {{ target_linux_git }} {{ active_linux_ref | default(target_linux_ref) }}"
@@ -50,7 +74,7 @@
   run_once: true
   delegate_to: localhost
   when:
-    - not bootlinux_tree_set_by_cli|bool
+    - needs_git_clone|bool
   tags: [ 'clone']
 
 - name: Fail if git ref does not exist
@@ -68,20 +92,11 @@
       - If using A/B testing with different refs, ensure shallow cloning is disabled
       - The repository URL '{{ target_linux_git }}' is correct and accessible
   when:
-    - not bootlinux_tree_set_by_cli|bool
+    - needs_git_clone|bool
     - ref_check.rc != 0
   run_once: true
   delegate_to: localhost
 
-- name: Check if target directory exists for dirty check
-  stat:
-    path: "{{ bootlinux_9p_host_path }}"
-  register: git_dir_stat
-  run_once: true
-  delegate_to: localhost
-  when:
-    - not bootlinux_tree_set_by_cli|bool
-
 - name: Check if git tree is dirty
   command: "git -C {{ bootlinux_9p_host_path }} status --porcelain"
   register: git_status
@@ -90,9 +105,9 @@
   run_once: true
   delegate_to: localhost
   when:
-    - not bootlinux_tree_set_by_cli|bool
-    - git_dir_stat.stat.exists
-    - git_dir_stat.stat.isdir
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
 
 - name: Fail if git tree has local modifications
   fail:
@@ -109,9 +124,9 @@
       Modified files:
       {{ git_status.stdout }}
   when:
-    - not bootlinux_tree_set_by_cli|bool
-    - git_dir_stat.stat.exists
-    - git_dir_stat.stat.isdir
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
     - git_status.stdout | length > 0
   run_once: true
   delegate_to: localhost
@@ -129,9 +144,65 @@
   until: not result.failed
   tags: [ 'clone']
   when:
-    - not bootlinux_tree_set_by_cli|bool
+    - needs_git_clone|bool
+  run_once: true
+  delegate_to: localhost
+
+- name: Get current git ref when git exists but clone wasn't needed
+  command: "git -C {{ bootlinux_9p_host_path }} rev-parse HEAD"
+  register: current_ref
+  changed_when: false
+  run_once: true
+  delegate_to: localhost
+  when:
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+
+- name: Get target ref SHA
+  command: "git -C {{ bootlinux_9p_host_path }} rev-parse {{ active_linux_ref | default(target_linux_ref) }}"
+  register: target_ref_sha
+  changed_when: false
+  failed_when: false
   run_once: true
   delegate_to: localhost
+  when:
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+
+- name: Fetch updates if target ref doesn't exist locally
+  command: "git -C {{ bootlinux_9p_host_path }} fetch origin"
+  run_once: true
+  delegate_to: localhost
+  when:
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+    - target_ref_sha.rc != 0
+
+- name: Get target ref SHA after fetch
+  command: "git -C {{ bootlinux_9p_host_path }} rev-parse {{ active_linux_ref | default(target_linux_ref) }}"
+  register: target_ref_sha_after_fetch
+  changed_when: false
+  run_once: true
+  delegate_to: localhost
+  when:
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+    - target_ref_sha.rc != 0
+
+- name: Checkout target ref if not on correct ref
+  command: "git -C {{ bootlinux_9p_host_path }} checkout {{ active_linux_ref | default(target_linux_ref) }}"
+  run_once: true
+  delegate_to: localhost
+  when:
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+    - (target_ref_sha.rc == 0 and current_ref.stdout != target_ref_sha.stdout) or
+      (target_ref_sha.rc != 0 and target_ref_sha_after_fetch is defined and target_ref_sha_after_fetch.rc == 0)
 
 - name: Copy kernel delta if requested on the control node
   template:
playbooks/roles/bootlinux/tasks/build/builder.yml +89 -9

133 lines not on this page · raw message

playbooks/roles/bootlinux/tasks/build/targets.yml +89 -9

134 lines not on this page · raw message

Signature · 1 line
2.47.2
#6 lore raw

PATCH 6/8 bootlinux: enhance A/B testing and repository management

In reply to Luis Chamberlain

Improve git repository handling and A/B testing capabilities:

- Add intelligent fallback logic for git ref resolution
- Handle both direct refs and tag resolution gracefully
- Improve error handling for missing or invalid refs
- Add better debugging output for git operations
- Update defconfig to enable 4K reflink testing
- Exclude linux directory from newline checking script

These changes make the bootlinux workflow more robust when dealing with different git ref types and improve the A/B testing experience with better error reporting and fallback mechanisms.

Generated-by Claude AI
---
 defconfigs/xfs_reflink_lbs                    |  1 +
 playbooks/roles/bootlinux/tasks/build/9p.yml  | 44 +++++++++++++++++--
 .../roles/bootlinux/tasks/build/builder.yml   | 39 ++++++++++++++--
 .../roles/bootlinux/tasks/build/targets.yml   | 39 ++++++++++++++--
 scripts/ensure_newlines.py                    |  3 +-
 5 files changed, 113 insertions(+), 13 deletions(-)
defconfigs/xfs_reflink_lbs +1 -0
index 77a94200..f20fa15a 100644
@@ -20,6 +20,7 @@ CONFIG_FSTESTS_XFS_ENABLE_LBS=y
 CONFIG_FSTESTS_XFS_ENABLE_LBS_4KS=y
 CONFIG_FSTESTS_XFS_SECTION_REFLINK_ENABLED=y
 
+CONFIG_FSTESTS_XFS_SECTION_REFLINK_4K=y
 CONFIG_FSTESTS_XFS_SECTION_REFLINK_8K_4KS=y
 CONFIG_FSTESTS_XFS_SECTION_REFLINK_16K_4KS=y
 CONFIG_FSTESTS_XFS_SECTION_REFLINK_32K_4KS=y
playbooks/roles/bootlinux/tasks/build/9p.yml +40 -4

72 lines not on this page · raw message

playbooks/roles/bootlinux/tasks/build/builder.yml +35 -4
index 1213c56f..014a341d 100644
@@ -137,24 +137,55 @@
     - git_directory_stat.stat.exists
     - target_ref_sha.rc != 0
 
-- name: Get target ref SHA after fetch
+- name: Try to resolve ref as direct ref after fetch
   command: "git -C {{ target_linux_dir_path }} rev-parse {{ target_linux_ref }}"
-  register: target_ref_sha_after_fetch
+  register: target_ref_sha_direct
   changed_when: false
+  failed_when: false
   when:
     - not needs_git_clone|bool
     - target_directory_stat.stat.exists
     - git_directory_stat.stat.exists
     - target_ref_sha.rc != 0
 
+- name: Try to resolve ref as remote branch if direct ref failed
+  command: "git -C {{ target_linux_dir_path }} rev-parse origin/{{ target_linux_ref }}"
+  register: target_ref_sha_remote
+  changed_when: false
+  failed_when: false
+  when:
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+    - target_ref_sha.rc != 0
+    - target_ref_sha_direct.rc != 0
+
+- name: Set resolved ref for checkout
+  set_fact:
+    resolved_ref: |
+      {%- if target_ref_sha.rc == 0 -%}
+      {{ target_linux_ref }}
+      {%- elif target_ref_sha_direct is defined and target_ref_sha_direct.rc == 0 -%}
+      {{ target_linux_ref }}
+      {%- elif target_ref_sha_remote is defined and target_ref_sha_remote.rc == 0 -%}
+      origin/{{ target_linux_ref }}
+      {%- else -%}
+      {{ target_linux_ref }}
+      {%- endif -%}
+  when:
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+
 - name: Checkout target ref if not on correct ref
-  command: "git -C {{ target_linux_dir_path }} checkout {{ target_linux_ref }}"
+  command: "git -C {{ target_linux_dir_path }} checkout {{ resolved_ref | default(target_linux_ref) }}"
   when:
     - not needs_git_clone|bool
     - target_directory_stat.stat.exists
     - git_directory_stat.stat.exists
     - (target_ref_sha.rc == 0 and current_ref.stdout != target_ref_sha.stdout) or
-      (target_ref_sha.rc != 0 and target_ref_sha_after_fetch is defined and target_ref_sha_after_fetch.rc == 0)
+      (target_ref_sha.rc != 0 and (target_ref_sha_direct is defined and target_ref_sha_direct.rc == 0)) or
+      (target_ref_sha.rc != 0 and (target_ref_sha_remote is defined and target_ref_sha_remote.rc == 0))
 
 - name: Copy the kernel delta to the builder
   ansible.builtin.template:
playbooks/roles/bootlinux/tasks/build/targets.yml +35 -4
index 87393c74..5942d2be 100644
@@ -139,24 +139,55 @@
     - git_directory_stat.stat.exists
     - target_ref_sha.rc != 0
 
-- name: Get target ref SHA after fetch
+- name: Try to resolve ref as direct ref after fetch
   command: "git -C {{ target_linux_dir_path }} rev-parse {{ target_linux_ref }}"
-  register: target_ref_sha_after_fetch
+  register: target_ref_sha_direct
   changed_when: false
+  failed_when: false
   when:
     - not needs_git_clone|bool
     - target_directory_stat.stat.exists
     - git_directory_stat.stat.exists
     - target_ref_sha.rc != 0
 
+- name: Try to resolve ref as remote branch if direct ref failed
+  command: "git -C {{ target_linux_dir_path }} rev-parse origin/{{ target_linux_ref }}"
+  register: target_ref_sha_remote
+  changed_when: false
+  failed_when: false
+  when:
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+    - target_ref_sha.rc != 0
+    - target_ref_sha_direct.rc != 0
+
+- name: Set resolved ref for checkout
+  set_fact:
+    resolved_ref: |
+      {%- if target_ref_sha.rc == 0 -%}
+      {{ target_linux_ref }}
+      {%- elif target_ref_sha_direct is defined and target_ref_sha_direct.rc == 0 -%}
+      {{ target_linux_ref }}
+      {%- elif target_ref_sha_remote is defined and target_ref_sha_remote.rc == 0 -%}
+      origin/{{ target_linux_ref }}
+      {%- else -%}
+      {{ target_linux_ref }}
+      {%- endif -%}
+  when:
+    - not needs_git_clone|bool
+    - target_directory_stat.stat.exists
+    - git_directory_stat.stat.exists
+
 - name: Checkout target ref if not on correct ref
-  command: "git -C {{ target_linux_dir_path }} checkout {{ target_linux_ref }}"
+  command: "git -C {{ target_linux_dir_path }} checkout {{ resolved_ref | default(target_linux_ref) }}"
   when:
     - not needs_git_clone|bool
     - target_directory_stat.stat.exists
     - git_directory_stat.stat.exists
     - (target_ref_sha.rc == 0 and current_ref.stdout != target_ref_sha.stdout) or
-      (target_ref_sha.rc != 0 and target_ref_sha_after_fetch is defined and target_ref_sha_after_fetch.rc == 0)
+      (target_ref_sha.rc != 0 and (target_ref_sha_direct is defined and target_ref_sha_direct.rc == 0)) or
+      (target_ref_sha.rc != 0 and (target_ref_sha_remote is defined and target_ref_sha_remote.rc == 0))
 
 - name: Copy kernel delta if requested on the target nodes
   template:
scripts/ensure_newlines.py +2 -1
index 969cf32a..f82ba136 100755
@@ -45,7 +45,8 @@ def main():
         dirs[:] = [
             d
             for d in dirs
-            if not d.startswith(".") and d not in ["__pycache__", "node_modules"]
+            if not d.startswith(".")
+            and d not in ["__pycache__", "node_modules", "linux"]
         ]
 
         for file in files:
Signature · 1 line
2.47.2
#7 lore raw

PATCH 7/8 fstests: add make target for running tests on all hosts

In reply to Luis Chamberlain

Add a new 'fstests-tests' make target that runs tests on both baseline and dev hosts simultaneously. This target is useful for A/B testing scenarios where you want to run the same tests on multiple host configurations in parallel.

The target uses the same FSTESTS_DYNAMIC_RUNTIME_VARS as the existing baseline and dev targets but limits execution to both baseline and dev host groups together.

Generated-by Claude AI
---
 workflows/fstests/Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)
workflows/fstests/Makefile +10 -0
index 77f8055c..64a9b61c 100644
@@ -198,6 +198,15 @@ fstests-dev: $(KDEVOPS_EXTRA_VARS)
 		'{ $(FSTESTS_DYNAMIC_RUNTIME_VARS) }' \
 		--extra-vars=@./extra_vars.yaml $(LIMIT_HOSTS)
 
+fstests-tests: $(KDEVOPS_EXTRA_VARS)
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+		--limit 'baseline:dev' \
+		playbooks/fstests.yml \
+		--tags vars,run_tests,copy_results \
+		--extra-vars \
+		'{ $(FSTESTS_DYNAMIC_RUNTIME_VARS) }' \
+		--extra-vars=@./extra_vars.yaml $(LIMIT_HOSTS)
+
 fstests-baseline-results-tfb-ls: $(KDEVOPS_EXTRA_VARS)
 	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
 		--limit 'baseline' \
@@ -249,6 +258,7 @@ fstests-help-menu:
 	@echo "fstests-kdevops-setup                   - Install kdevops specific files, the fstests and running test targets also runs this"
 	@echo "fstests-baseline                        - Run fstests on baseline hosts and collect results"
 	@echo "fstests-dev                             - Run fstests on dev hosts and collect results"
+	@echo "fstests-tests                           - Run fstests on both baseline and dev hosts simultaneously"
 	@echo ""
 	@echo "fstests-config                          - Generates the filesystem configuration file only onto target systems"
 	@echo "fstests-config-debug                    - Generates the filesystem configuration file locally, useful for debugging"
Signature · 1 line
2.47.2
#8 lore raw

PATCH 8/8 monitoring: integrate monitoring collection into fstests workflow

In reply to Luis Chamberlain

Fix monitoring data collection to work properly during fstests runs by changing from include_role to import_tasks. The include_role directive with tasks_from parameter wasn't executing properly due to tag filtering during playbook execution, preventing monitoring data from being collected.

Using import_tasks ensures the monitoring tasks are statically included at parse time and properly executed when the appropriate tags are present. This allows monitoring to run on all hosts and collect folio migration statistics during test execution.

The monitoring documentation has also been updated to reflect that it's now a shared service available to all workflows.

Generated-by Claude AI
---
 Kconfig                                |  4 ++++
 README.md                              |  1 +
 playbooks/roles/fstests/tasks/main.yml | 14 ++++++++++++++
 3 files changed, 19 insertions(+)
Kconfig +4 -0
index 988782a9..3f2bc8cf 100644
@@ -79,6 +79,10 @@ menu "Target workflows"
 source "kconfigs/workflows/Kconfig"
 endmenu
 
+menu "Monitors"
+source "kconfigs/monitors/Kconfig"
+endmenu
+
 menu "Kdevops configuration"
 source "kconfigs/Kconfig.kdevops"
 endmenu
README.md +1 -0
index e695d088..c9f44249 100644
@@ -306,6 +306,7 @@ Below is kdevops' recommended documentation reading.
   * [kdevops' evolving make help](docs/evolving-make-help.md)
   * [kdevops configuration](docs/kdevops-configuration.md)
   * [kdevops mirror support](docs/kdevops-mirror.md)
+  * [kdevops monitoring services](docs/monitoring.md)
   * [kdevops first run](docs/kdevops-first-run.md)
   * [kdevops running make](docs/running-make.md)
   * [kdevops libvirt storage pool considerations](docs/libvirt-storage-pool.md)
playbooks/roles/fstests/tasks/main.yml +14 -0
index 2665f693..3e2cbf99 100644
@@ -1235,6 +1235,13 @@
   when:
     - fstests_skip_run|bool
 
+# Start monitoring services before running tests
+- import_tasks: ../../monitoring/tasks/monitor_run.yml
+  when:
+    - kdevops_run_fstests|bool
+    - enable_monitoring|default(false)|bool
+  tags: [ 'oscheck', 'fstests', 'run_tests', 'monitoring', 'monitor_run' ]
+
 # Recent environments runs are showing that environment variables
 # set below are not propagated. So best to stuff what you need
 # into the .kdevops_fstests_setup file which is sourced by root.
@@ -1281,6 +1288,13 @@
   when:
     - kdevops_run_fstests|bool
 
+# Stop monitoring services and collect data after running tests
+- import_tasks: ../../monitoring/tasks/monitor_collect.yml
+  when:
+    - kdevops_run_fstests|bool
+    - enable_monitoring|default(false)|bool
+  tags: [ 'oscheck', 'fstests', 'run_tests', 'monitoring', 'monitor_collect' ]
+
 - name: Remove watchdog hint that tests have started
   local_action: file path="{{ fstests_workflow_dir }}/.begin" state=absent
   tags: [ 'oscheck', 'fstests', 'run_tests' ]
Signature · 1 line
2.47.2
#9 lore raw

Re: [PATCH 8/8] monitoring: integrate monitoring collection into fstests workflow

Quoted 10 lines · Luis Chamberlain

Fix monitoring data collection to work properly during fstests runs by changing from include_role to import_tasks. The include_role directive with tasks_from parameter wasn't executing properly due to tag filtering during playbook execution, preventing monitoring data from being collected.

Using import_tasks ensures the monitoring tasks are statically included at parse time and properly executed when the appropriate tags are present. This allows monitoring to run on all hosts and collect folio migration statistics during test execution.

Quoted 5 lines

The monitoring documentation has also been updated to reflect that it's now a shared service available to all workflows.

Generated-by Claude AI

Bah, this patch can be ignored for now, I forgot to git am a few things.

Luis

#10 lore raw

Re: [PATCH 8/8] monitoring: integrate monitoring collection into fstests workflow

Quoted 11 lines · Luis Chamberlain
Quoted 10 lines · Luis Chamberlain

Fix monitoring data collection to work properly during fstests runs by changing from include_role to import_tasks. The include_role directive with tasks_from parameter wasn't executing properly due to tag filtering during playbook execution, preventing monitoring data from being collected.

Using import_tasks ensures the monitoring tasks are statically included at parse time and properly executed when the appropriate tags are present. This allows monitoring to run on all hosts and collect folio migration statistics during test execution.

Quoted 7 lines
Quoted 5 lines

The monitoring documentation has also been updated to reflect that it's now a shared service available to all workflows.

Generated-by Claude AI

Bah, this patch can be ignored for now, I forgot to git am a few things.

OK here's a v2 of that patch follows and a branch with *all* my pending changes:

https://github.com/linux-kdevops/kdevops/tree/20250811-monitor-v2

From 75bbc91c5cdaced1ab294b979cf58c13d8ead34c Mon Sep 17 00:00:00 2001
From: Luis Chamberlain <[email protected]>
Date: Mon, 11 Aug 2025 15:33:17 -0700
Subject: [PATCH v2] monitoring: add monitoring framework for workflow execution

Add a flexible monitoring framework that collects system metrics during workflow execution. The framework supports background monitoring services that automatically start before workflows and collect results afterward.

Initial implementation includes:
- Core monitoring infrastructure with Kconfig integration
- Folio migration statistics monitor (for developmental kernel features)
- Integration with fstests workflow
- Result collection and visualization support
- Documentation for adding new monitors and integrating with workflows

The monitoring system is designed to be modular, allowing workflows to opt-in and new monitors to be easily added. Results are stored in workflow-specific directories and can include both raw data and visualizations.

Generated-by Claude AI
---
 Kconfig                                       |   4 +
 README.md                                     |   1 +
 docs/monitoring.md                            | 279 ++++++++++++++++++
 kconfigs/monitors/Kconfig                     |  74 +++++
 playbooks/roles/fstests/tasks/main.yml        |  14 +
 playbooks/roles/monitoring/defaults/main.yml  |  18 ++
 .../monitoring/files/plot_migration_stats.py  | 220 ++++++++++++++
 playbooks/roles/monitoring/tasks/main.yml     |  23 ++
 .../monitoring/tasks/monitor_collect.yml      | 205 +++++++++++++
 .../roles/monitoring/tasks/monitor_run.yml    |  83 ++++++
 10 files changed, 921 insertions(+)
 create mode 100644 docs/monitoring.md
 create mode 100644 kconfigs/monitors/Kconfig
 create mode 100644 playbooks/roles/monitoring/defaults/main.yml
 create mode 100755 playbooks/roles/monitoring/files/plot_migration_stats.py
 create mode 100644 playbooks/roles/monitoring/tasks/main.yml
 create mode 100644 playbooks/roles/monitoring/tasks/monitor_collect.yml
 create mode 100644 playbooks/roles/monitoring/tasks/monitor_run.yml
Kconfig +4 -0
index 988782a9dc83..3f2bc8cf019a 100644
@@ -79,6 +79,10 @@ menu "Target workflows"
 source "kconfigs/workflows/Kconfig"
 endmenu
 
+menu "Monitors"
+source "kconfigs/monitors/Kconfig"
+endmenu
+
 menu "Kdevops configuration"
 source "kconfigs/Kconfig.kdevops"
 endmenu
README.md +1 -0
index e695d088dcb8..c9f442492559 100644
@@ -306,6 +306,7 @@ Below is kdevops' recommended documentation reading.
   * [kdevops' evolving make help](docs/evolving-make-help.md)
   * [kdevops configuration](docs/kdevops-configuration.md)
   * [kdevops mirror support](docs/kdevops-mirror.md)
+  * [kdevops monitoring services](docs/monitoring.md)
   * [kdevops first run](docs/kdevops-first-run.md)
   * [kdevops running make](docs/running-make.md)
   * [kdevops libvirt storage pool considerations](docs/libvirt-storage-pool.md)
docs/monitoring.md new file +279 -0

279 lines not on this page · raw message

kconfigs/monitors/Kconfig new file +74 -0

74 lines not on this page · raw message

playbooks/roles/fstests/tasks/main.yml +14 -0
index 2665f693af3c..3e2cbf99ff40 100644
@@ -1235,6 +1235,13 @@
   when:
     - fstests_skip_run|bool
 
+# Start monitoring services before running tests
+- import_tasks: ../../monitoring/tasks/monitor_run.yml
+  when:
+    - kdevops_run_fstests|bool
+    - enable_monitoring|default(false)|bool
+  tags: [ 'oscheck', 'fstests', 'run_tests', 'monitoring', 'monitor_run' ]
+
 # Recent environments runs are showing that environment variables
 # set below are not propagated. So best to stuff what you need
 # into the .kdevops_fstests_setup file which is sourced by root.
@@ -1281,6 +1288,13 @@
   when:
     - kdevops_run_fstests|bool
 
+# Stop monitoring services and collect data after running tests
+- import_tasks: ../../monitoring/tasks/monitor_collect.yml
+  when:
+    - kdevops_run_fstests|bool
+    - enable_monitoring|default(false)|bool
+  tags: [ 'oscheck', 'fstests', 'run_tests', 'monitoring', 'monitor_collect' ]
+
 - name: Remove watchdog hint that tests have started
   local_action: file path="{{ fstests_workflow_dir }}/.begin" state=absent
   tags: [ 'oscheck', 'fstests', 'run_tests' ]
playbooks/roles/monitoring/defaults/main.yml new file +18 -0
new file mode 100644
index 000000000000..7306f5959859
@@ -0,0 +1,18 @@
+---
+# Default values for monitoring role
+
+# Enable monitoring services
+enable_monitoring: false
+
+# Enable developmental statistics
+monitor_developmental_stats: false
+
+# Enable folio migration monitoring
+monitor_folio_migration: false
+
+# Folio migration monitoring interval in seconds
+monitor_folio_migration_interval: 60
+
+# Base path to store monitoring results on the control host
+# Workflows can override by setting monitoring_results_base_path
+monitoring_results_base_path: "{{ topdir_path }}/workflows/fstests/results/monitoring"
playbooks/roles/monitoring/files/plot_migration_stats.py new file +220 -0

220 lines not on this page · raw message

playbooks/roles/monitoring/tasks/main.yml new file +23 -0
new file mode 100644
index 000000000000..9c4bd3ab04e1
@@ -0,0 +1,23 @@
+---
+- name: Import optional extra_args file
+  include_vars: "{{ item }}"
+  ignore_errors: yes
+  with_first_found:
+    - files:
+      - "../extra_vars.yml"
+      - "../extra_vars.yaml"
+      - "../extra_vars.json"
+      skip: true
+  tags: vars
+
+- name: Include monitor_run tasks
+  include_tasks: monitor_run.yml
+  when:
+    - enable_monitoring|default(false)|bool
+  tags: [ 'monitoring', 'monitor_run' ]
+
+- name: Include monitor_collect tasks
+  include_tasks: monitor_collect.yml
+  when:
+    - enable_monitoring|default(false)|bool
+  tags: [ 'monitoring', 'monitor_collect' ]
playbooks/roles/monitoring/tasks/monitor_collect.yml new file +205 -0

205 lines not on this page · raw message

playbooks/roles/monitoring/tasks/monitor_run.yml new file +83 -0

83 lines not on this page · raw message

Signature · 1 line
2.45.2
#11 lore raw

Re: [PATCH 8/8] monitoring: integrate monitoring collection into fstests workflow

Quoted 18 lines · Luis Chamberlain
Quoted 17 lines · Luis Chamberlain
Quoted 15 lines · Luis Chamberlain

Fix monitoring data collection to work properly during fstests runs by changing from include_role to import_tasks. The include_role directive with tasks_from parameter wasn't executing properly due to tag filtering during playbook execution, preventing monitoring data from being collected.

Using import_tasks ensures the monitoring tasks are statically included at parse time and properly executed when the appropriate tags are present. This allows monitoring to run on all hosts and collect folio migration statistics during test execution.

The monitoring documentation has also been updated to reflect that it's now a shared service available to all workflows.

Generated-by Claude AI
Quoted 6 lines
Quoted 1 line

Bah, this patch can be ignored for now, I forgot to git am a few things.

OK here's a v2 of that patch follows and a branch with *all* my pending changes:

https://github.com/linux-kdevops/kdevops/tree/20250811-monitor-v2

I've pushed a better version of this, along with some fstests watchdog enhancements.

Luis