Skip to contentSkip to conversation

declared hosts support

PATCH In review Luis Chamberlain 1 participant 4 messages Aug 29, 2025 Kbuild

lore
Share & tools
lore
#0 lore raw

cover declared hosts support

It should be easy for us to add bare metal support because well, the biggest pain is bring up. If bringup and ssh is already configured, then all we have to do is just skip a bunch of things. So the question to ask, is how do we generate the hosts file? The easiest interface I could think of is to allow you to specify it at make menuconfig time or defconfig time with a DECLARE_HOSTS= specifier with either a space or comma separated lists of hosts.

I tested this with a new workflow I am testing, minio WARP testing, but I also tested this simply with a random host I can ssh to with just:

make defconfig-xfs_reflink_4k DECLARE_HOSTS=foo ; make AV=2 make uname

And that works. For my other workflow I am testing I got full test restults and mkfs'd a drive. And so each workflow needs a bit of love for further DECLARE_HOSTS support. Things like the data partition, and so on. I suspect we can just specify some of these in the command line as CLI variables.

But this goes with just basic raw support. Maybe we only enable workflows for KDEVOPS_USE_DECLARED_HOSTS if we declare the workflow as supporting it (maybe select SUPPORTS_DECLARED_HOSTS)?

Luis Chamberlain (2):
gen_hosts: refactor monolithic hosts.j2 template into per-workflow
templates
declared_hosts: add support for pre-existing infrastructure

 Makefile                                      |   7 +
 kconfigs/Kconfig.bringup                      |   7 +
 kconfigs/Kconfig.declared_hosts               |  65 +++++
 playbooks/roles/devconfig/tasks/main.yml      |  60 ++++
 playbooks/roles/gen_hosts/tasks/main.yml      |  40 ++-
 playbooks/roles/gen_hosts/templates/hosts.j2  | 264 +++---------------
 .../roles/gen_hosts/templates/workflows/ai.j2 |  99 +++++++
 .../gen_hosts/templates/workflows/blktests.j2 |  58 ++++
 .../gen_hosts/templates/workflows/cxl.j2      |   7 +
 .../templates/workflows/declared-hosts.j2     | 219 +++++++++++++++
 .../gen_hosts/templates/workflows/default.j2  |  62 ++++
 .../templates/workflows/fio-tests.j2          |  38 +++
 .../gen_hosts/templates/workflows/fstests.j2  |  72 +++++
 .../gen_hosts/templates/workflows/gitr.j2     |  41 +++
 .../gen_hosts/templates/workflows/ltp.j2      |  41 +++
 .../gen_hosts/templates/workflows/mmtests.j2  |  77 +++++
 .../gen_hosts/templates/workflows/nfstest.j2  |  41 +++
 .../gen_hosts/templates/workflows/pynfs.j2    |   7 +
 .../templates/workflows/reboot-limit.j2       |  33 +++
 .../templates/workflows/selftests.j2          |  53 ++++
 .../gen_hosts/templates/workflows/sysbench.j2 |  53 ++++
 playbooks/roles/gen_nodes/tasks/main.yml      |   5 +
 workflows/ai/Makefile                         |   3 +-
 workflows/blktests/Makefile                   |   3 +-
 workflows/cxl/Makefile                        |   3 +-
 workflows/fio-tests/Makefile                  |   3 +-
 workflows/fstests/Makefile                    |   3 +-
 workflows/gitr/Makefile                       |   3 +-
 workflows/linux/Makefile                      |   3 +-
 workflows/ltp/Makefile                        |   3 +-
 workflows/mmtests/Makefile                    |   3 +-
 workflows/nfstest/Makefile                    |   3 +-
 workflows/pynfs/Makefile                      |   3 +-
 workflows/selftests/Makefile                  |   3 +-
 workflows/sysbench/Makefile                   |   3 +-
 35 files changed, 1143 insertions(+), 245 deletions(-)
 create mode 100644 kconfigs/Kconfig.declared_hosts
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ai.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/blktests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/cxl.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/declared-hosts.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/default.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fio-tests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fstests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/gitr.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ltp.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mmtests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/nfstest.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/pynfs.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/reboot-limit.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/selftests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/sysbench.j2
Signature · 1 line
2.50.1
#1 lore raw

PATCH 1/2 gen_hosts: refactor monolithic hosts.j2 template into per-workflow templates

The hosts.j2 template had become unwieldy spaghetti code with all workflows mixed together in a single file, making it difficult to read, maintain, and extend. This was particularly problematic when attempting to add new features.

This refactoring:
- Splits the monolithic hosts.j2 into individual workflow templates under playbooks/roles/gen_hosts/templates/workflows/
- Creates dedicated templates for each workflow (fstests, blktests, selftests, mmtests, sysbench, ai, fio-tests, etc.)
- Converts the main hosts.j2 to use Jinja2 includes for cleaner organization
- Maintains exact same functionality with improved maintainability
- Removes workflow-specific template overrides from all workflow Makefiles as they now all use the unified hosts.j2 template

Benefits:
- Each workflow template is now self-contained and easy to understand
- Adding new workflows no longer requires navigating through hundreds of lines of unrelated configuration
- Template logic is easier to debug and test
- All workflows now use a consistent template system

This is a pure refactoring with no functional changes. The generated hosts files remain identical to the previous implementation.

Generated-by Claude AI
---
 playbooks/roles/gen_hosts/tasks/main.yml      |  40 ++-
 playbooks/roles/gen_hosts/templates/hosts.j2  | 264 +++---------------
 .../roles/gen_hosts/templates/workflows/ai.j2 |  99 +++++++
 .../gen_hosts/templates/workflows/blktests.j2 |  58 ++++
 .../gen_hosts/templates/workflows/cxl.j2      |   7 +
 .../gen_hosts/templates/workflows/default.j2  |  62 ++++
 .../templates/workflows/fio-tests.j2          |  38 +++
 .../gen_hosts/templates/workflows/fstests.j2  |  72 +++++
 .../gen_hosts/templates/workflows/gitr.j2     |  41 +++
 .../gen_hosts/templates/workflows/ltp.j2      |  41 +++
 .../gen_hosts/templates/workflows/mmtests.j2  |  77 +++++
 .../gen_hosts/templates/workflows/nfstest.j2  |  41 +++
 .../gen_hosts/templates/workflows/pynfs.j2    |   7 +
 .../templates/workflows/reboot-limit.j2       |  33 +++
 .../templates/workflows/selftests.j2          |  53 ++++
 .../gen_hosts/templates/workflows/sysbench.j2 |  53 ++++
 workflows/ai/Makefile                         |   3 +-
 workflows/blktests/Makefile                   |   3 +-
 workflows/cxl/Makefile                        |   3 +-
 workflows/fio-tests/Makefile                  |   3 +-
 workflows/fstests/Makefile                    |   3 +-
 workflows/gitr/Makefile                       |   3 +-
 workflows/linux/Makefile                      |   3 +-
 workflows/ltp/Makefile                        |   3 +-
 workflows/mmtests/Makefile                    |   3 +-
 workflows/nfstest/Makefile                    |   3 +-
 workflows/pynfs/Makefile                      |   3 +-
 workflows/selftests/Makefile                  |   3 +-
 workflows/sysbench/Makefile                   |   3 +-
 29 files changed, 780 insertions(+), 245 deletions(-)
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ai.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/blktests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/cxl.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/default.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fio-tests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/fstests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/gitr.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/ltp.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/mmtests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/nfstest.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/pynfs.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/reboot-limit.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/selftests.j2
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/sysbench.j2
playbooks/roles/gen_hosts/tasks/main.yml +39 -1
index fb63629a..657fd22b 100644
@@ -10,6 +10,7 @@
       skip: true
   tags: vars
 
+
 - name: Get our user
   ansible.builtin.command: "whoami"
   register: my_user
@@ -39,6 +40,21 @@
     path: "{{ kdevops_hosts_template_full_path }}"
   register: ansible_hosts_template
 
+- name: Generate the Ansible hosts file for declared hosts configuration
+  tags: ['hosts']
+  ansible.builtin.template:
+    src: "{{ kdevops_hosts_template }}"
+    dest: "{{ ansible_cfg_inventory }}"
+    force: true
+    trim_blocks: True
+    lstrip_blocks: True
+    mode: '0644'
+  when:
+    - kdevops_use_declared_hosts|default(false)|bool
+    - kdevops_declared_hosts is defined
+    - kdevops_declared_hosts|length > 0
+    - ansible_hosts_template.stat.exists
+
 - name: Set fstests config file variable for {{ fstests_fstyp }}
   ansible.builtin.set_fact:
     is_fstests: true
@@ -67,6 +83,7 @@
   when:
     - bootlinux_builder
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file
   tags: ["hosts"]
@@ -80,6 +97,7 @@
     - not kdevops_workflows_dedicated_workflow
     - ansible_hosts_template.stat.exists
     - not kdevops_enable_nixos|default(false)|bool
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for NixOS
   tags: ['hosts']
@@ -93,6 +111,7 @@
     - not kdevops_workflows_dedicated_workflow
     - ansible_hosts_template.stat.exists
     - kdevops_enable_nixos|default(false)|bool
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Update Ansible inventory access modification time so make sees it updated
   ansible.builtin.file:
@@ -112,6 +131,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_cxl
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for dedicated pynfs work
   tags: ["hosts"]
@@ -125,6 +145,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_pynfs
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for dedicated gitr workflow
   tags: ["hosts"]
@@ -140,6 +161,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_gitr
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate an Ansible inventory file for a dedicated ltp workflow
   tags: ["hosts"]
@@ -155,6 +177,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_ltp
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for dedicated nfstest workflow
   tags: ["hosts"]
@@ -170,6 +193,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_nfstest
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Set empty enabled test types list for fstests
   tags: ["hosts"]
@@ -178,6 +202,7 @@
   when:
     - is_fstests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Check which fstests test types are enabled
   register: fstests_enabled_test_types_reg
@@ -202,6 +227,7 @@
   when:
     - is_fstests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Now expand the list of enabled fstests for valid configuration sections
   tags: ["hosts"]
@@ -213,6 +239,7 @@
   when:
     - is_fstests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
     - item.changed
 
 - name: Generate the Ansible inventory file for a dedicated fstests setup
@@ -233,6 +260,7 @@
   when:
     - is_fstests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Infer enabled blktests test section types
   ansible.builtin.set_fact:
@@ -247,6 +275,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_blktests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Debug inferring block test types
   ansible.builtin.debug:
@@ -269,6 +298,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_blktests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Infer enabled selftests test section types
   ansible.builtin.set_fact:
@@ -283,6 +313,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_selftests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for a dedicated selftests setup
   tags: ["hosts"]
@@ -296,6 +327,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_selftests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Collect dynamically supported filesystems
   vars:
@@ -338,6 +370,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_sysbench
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 
 - name: Generate the Ansible hosts file for a dedicated fio-tests setup
@@ -354,6 +387,7 @@
     - kdevops_workflow_enable_fio_tests
     - ansible_hosts_template.stat.exists
     - not kdevops_enable_nixos|default(false)|bool
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 
 - name: Infer enabled mmtests test types
@@ -369,6 +403,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_mmtests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible hosts file for a dedicated mmtests setup
   tags: ["hosts"]
@@ -382,6 +417,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_mmtests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible hosts file for a dedicated reboot-limit setup
   tags: ["hosts"]
@@ -395,6 +431,7 @@
     - kdevops_workflows_dedicated_workflow
     - workflows_reboot_limit
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Load AI nodes configuration for multi-filesystem setup
   include_vars:
@@ -426,8 +463,9 @@
     mode: '0644'
   when:
     - kdevops_workflows_dedicated_workflow
-    - kdevops_workflow_enable_ai
+    - kdevops_workflow_enable_ai|default(false)|bool
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Verify if final host file exists
   ansible.builtin.stat:
playbooks/roles/gen_hosts/templates/hosts.j2 +33 -231

276 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/ai.j2 new file +99 -0

99 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/blktests.j2 new file +58 -0

58 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/cxl.j2 new file +7 -0
new file mode 100644
index 00000000..53790f29
@@ -0,0 +1,7 @@
+{# Workflow template for CXL #}
+[all]
+localhost ansible_connection=local
+write-your-own-template-for-cxl-workflow
+
+[all:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/workflows/default.j2 new file +62 -0

62 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/fio-tests.j2 new file +38 -0
new file mode 100644
index 00000000..548941a0
@@ -0,0 +1,38 @@
+{# Workflow template for fio-tests #}
+[all]
+localhost ansible_connection=local
+{{ kdevops_host_prefix }}-fio-tests
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-fio-tests-dev
+{% endif %}
+
+[all:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+[baseline]
+{{ kdevops_host_prefix }}-fio-tests
+
+[baseline:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+{% if kdevops_baseline_and_dev %}
+[dev]
+{{ kdevops_host_prefix }}-fio-tests-dev
+
+[dev:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+{% endif %}
+[fio_tests]
+{{ kdevops_host_prefix }}-fio-tests
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-fio-tests-dev
+{% endif %}
+
+[fio_tests:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+[service]
+
+[service:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/workflows/fstests.j2 new file +72 -0

72 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/gitr.j2 new file +41 -0

41 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/ltp.j2 new file +41 -0

41 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/mmtests.j2 new file +77 -0

77 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/nfstest.j2 new file +41 -0

41 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/pynfs.j2 new file +7 -0
new file mode 100644
index 00000000..6145b1a0
@@ -0,0 +1,7 @@
+{# Workflow template for pynfs #}
+[all]
+localhost ansible_connection=local
+write-your-own-template-for-pynfs-workflow
+
+[all:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/workflows/reboot-limit.j2 new file +33 -0
new file mode 100644
index 00000000..07bd6f80
@@ -0,0 +1,33 @@
+{# Workflow template for reboot-limit #}
+[all]
+localhost ansible_connection=local
+{{ kdevops_host_prefix }}-reboot-limit
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-reboot-limit-dev
+{% endif %}
+
+[all:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+[baseline]
+{{ kdevops_host_prefix }}-reboot-limit
+
+[baseline:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+{% if kdevops_baseline_and_dev %}
+[dev]
+{{ kdevops_host_prefix }}-reboot-limit-dev
+
+[dev:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+
+{% endif %}
+[reboot-limit]
+{{ kdevops_host_prefix }}-reboot-limit
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_host_prefix }}-reboot-limit-dev
+{% endif %}
+
+[reboot-limit:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_hosts/templates/workflows/selftests.j2 new file +53 -0

53 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/sysbench.j2 new file +53 -0

53 lines not on this page · raw message

workflows/ai/Makefile +2 -1
index 1c297edd..27dea382 100644
@@ -4,7 +4,8 @@ PHONY += ai-tests ai-tests-baseline ai-tests-dev
 PHONY += ai-tests-results
 
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := hosts.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := hosts.j2
 endif
 
 export AI_DATA_TARGET := $(subst ",,$(CONFIG_AI_BENCHMARK_RESULTS_DIR))
workflows/blktests/Makefile +2 -1
index 11cd9a65..97649026 100644
@@ -12,7 +12,8 @@ ID=$(shell id -u)
 BLKTESTS_ARGS	:=
 
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := blktests.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := blktests.j2
 endif
 
 BLKTESTS_GIT:=$(subst ",,$(CONFIG_BLKTESTS_GIT))
workflows/cxl/Makefile +2 -1
index bb130057..feabe197 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: copyleft-next-0.3.1
 
-export KDEVOPS_HOSTS_TEMPLATE := cxl.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := cxl.j2
 
 CXL_ARGS :=
 CXL_ARGS += ndctl_git='$(subst ",,$(CONFIG_NDCTL_GIT))'
workflows/fio-tests/Makefile +2 -1
index 5eb2ccbd..20a5d192 100644
@@ -1,5 +1,6 @@
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := fio-tests.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := fio-tests.j2
 endif
 
 fio-tests:
workflows/fstests/Makefile +2 -1
index d2e5c636..989c564b 100644
@@ -9,7 +9,8 @@ FSTESTS_BASELINE_EXTRA :=
 export FSTYP:=$(subst ",,$(CONFIG_FSTESTS_FSTYP))
 
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := fstests.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := fstests.j2
 endif
 
 FSTESTS_ARGS += fstests_fstyp='$(FSTYP)'
workflows/gitr/Makefile +2 -1
index c685395a..eadcdac1 100644
@@ -1,5 +1,6 @@
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := gitr.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := gitr.j2
 endif
 
 GITR_MNT:=$(subst ",,$(CONFIG_GITR_MNT))
workflows/linux/Makefile +2 -1
index 30b123f9..60d755b3 100644
@@ -14,7 +14,8 @@ TREE_CONFIG:=config-$(TREE_REF)-pure-iomap
 endif
 
 ifeq (y,$(CONFIG_BOOTLINUX_BUILDER))
-KDEVOPS_HOSTS_TEMPLATE=builder.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# KDEVOPS_HOSTS_TEMPLATE=builder.j2
 endif
 
 # Describes the Linux clone
workflows/ltp/Makefile +2 -1
index 767465bc..b3c3319b 100644
@@ -1,5 +1,6 @@
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := ltp.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := ltp.j2
 endif
 
 LTP_REPO:=$(subst ",,$(CONFIG_LTP_REPO))
workflows/mmtests/Makefile +2 -1
index b65d256b..186dc8cb 100644
@@ -1,7 +1,8 @@
 MMTESTS_ARGS	:=
 
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := mmtests.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := mmtests.j2
 endif
 
 mmtests:
workflows/nfstest/Makefile +2 -1
index bbfd3f64..bc880b6d 100644
@@ -1,5 +1,6 @@
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := nfstest.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := nfstest.j2
 endif # CONFIG_WORKFLOWS_DEDICATED_WORKFLOW
 
 ifeq (y,$(CONFIG_NFSTEST_USE_KDEVOPS_NFSD))
workflows/pynfs/Makefile +2 -1
index 2f3ff97b..b7bb12bb 100644
@@ -1,5 +1,6 @@
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := pynfs.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := pynfs.j2
 endif
 
 PYNFS_GIT:=$(subst ",,$(CONFIG_PYNFS_GIT))
workflows/selftests/Makefile +2 -1
index d3b7044c..4ec691f3 100644
@@ -3,7 +3,8 @@
 SELFTESTS_ARGS :=
 
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := selftests.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := selftests.j2
 endif
 
 SELFTESTS_DYNAMIC_RUNTIME_VARS := "kdevops_run_selftests": True
workflows/sysbench/Makefile +2 -1
index daf7bc75..5f484691 100644
@@ -1,7 +1,8 @@
 PHONY += sysbench sysbench-test sysbench-telemetry sysbench-help-menu
 
 ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
-export KDEVOPS_HOSTS_TEMPLATE := sysbench.j2
+# After refactoring, all workflows use hosts.j2 which includes workflow-specific templates
+# export KDEVOPS_HOSTS_TEMPLATE := sysbench.j2
 endif
 
 TAGS_SYSBENCH_RUN := db_start
Signature · 1 line
2.50.1
#2 lore raw

Re: [PATCH 1/2] gen_hosts: refactor monolithic hosts.j2 template into per-workflow templates

Quoted 169 lines · Luis Chamberlain
playbooks/roles/gen_hosts/tasks/main.yml +21 -0
@@ -67,6 +83,7 @@
   when:
     - bootlinux_builder
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file
   tags: ["hosts"]
@@ -80,6 +97,7 @@
     - not kdevops_workflows_dedicated_workflow
     - ansible_hosts_template.stat.exists
     - not kdevops_enable_nixos|default(false)|bool
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for NixOS
   tags: ['hosts']
@@ -93,6 +111,7 @@
     - not kdevops_workflows_dedicated_workflow
     - ansible_hosts_template.stat.exists
     - kdevops_enable_nixos|default(false)|bool
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Update Ansible inventory access modification time so make sees it updated
   ansible.builtin.file:
@@ -112,6 +131,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_cxl
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for dedicated pynfs work
   tags: ["hosts"]
@@ -125,6 +145,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_pynfs
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for dedicated gitr workflow
   tags: ["hosts"]
@@ -140,6 +161,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_gitr
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate an Ansible inventory file for a dedicated ltp workflow
   tags: ["hosts"]
@@ -155,6 +177,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_ltp
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for dedicated nfstest workflow
   tags: ["hosts"]
@@ -170,6 +193,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_nfstest
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Set empty enabled test types list for fstests
   tags: ["hosts"]
@@ -178,6 +202,7 @@
   when:
     - is_fstests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Check which fstests test types are enabled
   register: fstests_enabled_test_types_reg
@@ -202,6 +227,7 @@
   when:
     - is_fstests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Now expand the list of enabled fstests for valid configuration sections
   tags: ["hosts"]
@@ -213,6 +239,7 @@
   when:
     - is_fstests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
     - item.changed
 
 - name: Generate the Ansible inventory file for a dedicated fstests setup
@@ -233,6 +260,7 @@
   when:
     - is_fstests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Infer enabled blktests test section types
   ansible.builtin.set_fact:
@@ -247,6 +275,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_blktests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Debug inferring block test types
   ansible.builtin.debug:
@@ -269,6 +298,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_blktests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Infer enabled selftests test section types
   ansible.builtin.set_fact:
@@ -283,6 +313,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_selftests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible inventory file for a dedicated selftests setup
   tags: ["hosts"]
@@ -296,6 +327,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_selftests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Collect dynamically supported filesystems
   vars:
@@ -338,6 +370,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_sysbench
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 
 - name: Generate the Ansible hosts file for a dedicated fio-tests setup
@@ -354,6 +387,7 @@
     - kdevops_workflow_enable_fio_tests
     - ansible_hosts_template.stat.exists
     - not kdevops_enable_nixos|default(false)|bool
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 
 - name: Infer enabled mmtests test types
@@ -369,6 +403,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_mmtests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible hosts file for a dedicated mmtests setup
   tags: ["hosts"]
@@ -382,6 +417,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_mmtests
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Generate the Ansible hosts file for a dedicated reboot-limit setup
   tags: ["hosts"]
@@ -395,6 +431,7 @@
     - kdevops_workflows_dedicated_workflow
     - workflows_reboot_limit
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 
 - name: Load AI nodes configuration for multi-filesystem setup
   include_vars:

playbooks/roles/gen_hosts/tasks/main.yml line 469

Quoted 8 lines
playbooks/roles/gen_hosts/tasks/main.yml +2 -1
@@ -426,8 +463,9 @@
     mode: '0644'
   when:
     - kdevops_workflows_dedicated_workflow
-    - kdevops_workflow_enable_ai
+    - kdevops_workflow_enable_ai|default(false)|bool
     - ansible_hosts_template.stat.exists
+    - not kdevops_use_declared_hosts|default(false)|bool
 

Sorry, these hunks should have been part of the second patch.

Luis

#3 lore raw

PATCH 2/2 declared_hosts: add support for pre-existing infrastructure

In reply to Luis Chamberlain

This adds support for using pre-existing infrastructure (bare metal servers, pre-provisioned VMs) that already have SSH access configured. This is useful for environments where kdevops bringup/teardown is not desired or possible.

Key features:
- New DECLARE_HOSTS CLI parameter to specify existing hosts
- Automatic detection of CLI override to enable declared hosts mode
- SKIP_BRINGUP is automatically selected when using declared hosts
- Hosts can be specified as space or comma-separated list
- For A/B testing workflows, odd-numbered hosts become baseline nodes and even-numbered hosts become dev nodes
- New Kconfig.declared_hosts for configuration options
- New declared-hosts.j2 template for generating inventory

Usage:
make defconfig-xfs_reflink_4k DECLARE_HOSTS='server1 server2 server3' make # Will skip bringup and use the declared hosts

The implementation assumes SSH access is already configured in the user's ~/.ssh/config or via SSH keys. No SSH configuration is generated by kdevops when using declared hosts.

This feature integrates cleanly with the refactored template system, using the same hosts.j2 entry point with a dedicated declared-hosts.j2 template for this mode.

Generated-by Claude AI
---
 Makefile                                      |   7 +
 kconfigs/Kconfig.bringup                      |   7 +
 kconfigs/Kconfig.declared_hosts               |  65 ++++++
 playbooks/roles/devconfig/tasks/main.yml      |  60 +++++
 .../templates/workflows/declared-hosts.j2     | 219 ++++++++++++++++++
 playbooks/roles/gen_nodes/tasks/main.yml      |   5 +
 6 files changed, 363 insertions(+)
 create mode 100644 kconfigs/Kconfig.declared_hosts
 create mode 100644 playbooks/roles/gen_hosts/templates/workflows/declared-hosts.j2
Makefile +7 -0
index 83c67340..fcb626ae 100644
@@ -19,6 +19,13 @@ export KDEVOPS_NODES :=
 export PYTHONUNBUFFERED=1
 export TOPDIR=./
 export TOPDIR_PATH = $(shell readlink -f $(TOPDIR))
+
+# Export CLI override variables for Kconfig to detect them
+# Note: We accept DECLARE_HOSTS but export as DECLARED_HOSTS for consistency
+ifdef DECLARE_HOSTS
+export DECLARED_HOSTS := $(DECLARE_HOSTS)
+endif
+
 include scripts/refs.Makefile
 
 KDEVOPS_NODES_ROLE_TEMPLATE_DIR :=		$(KDEVOPS_PLAYBOOKS_DIR)/roles/gen_nodes/templates
kconfigs/Kconfig.bringup +7 -0
index 8caf07be..f7f9735a 100644
@@ -9,8 +9,14 @@ config KDEVOPS_ENABLE_NIXOS
 	bool
 	output yaml
 
+# CLI override detection for DECLARED_HOSTS which should enable SKIP_BRINGUP
+config SKIP_BRINGUP_SET_BY_CLI
+	bool
+	default $(shell, scripts/check-cli-set-var.sh DECLARED_HOSTS)
+
 choice
 	prompt "Node bring up method"
+	default SKIP_BRINGUP if SKIP_BRINGUP_SET_BY_CLI
 	default GUESTFS
 
 config GUESTFS
@@ -80,6 +86,7 @@ config LIBVIRT
 source "kconfigs/Kconfig.guestfs"
 source "kconfigs/Kconfig.nixos"
 source "terraform/Kconfig"
+source "kconfigs/Kconfig.declared_hosts"
 if LIBVIRT
 source "kconfigs/Kconfig.libvirt"
 endif
kconfigs/Kconfig.declared_hosts new file +65 -0

65 lines not on this page · raw message

playbooks/roles/devconfig/tasks/main.yml +60 -0

66 lines not on this page · raw message

playbooks/roles/gen_hosts/templates/workflows/declared-hosts.j2 new file +219 -0

219 lines not on this page · raw message

playbooks/roles/gen_nodes/tasks/main.yml +5 -0
index b1a1946f..0fa778bf 100644
@@ -20,6 +20,11 @@
   register: my_group
   changed_when: false
 
+- name: Skip node generation if using declared hosts
+  meta: end_play
+  when:
+    - kdevops_use_declared_hosts|default(false)|bool
+
 - name: Create guestfs directory
   ansible.builtin.file:
     path: "{{ guestfs_path }}"
Signature · 1 line
2.50.1