Skip to contentSkip to conversation

Build once, test everywhere

RFC PATCH In review [email protected] 3 participants 9 messages Apr 22, 2025 – Apr 24, 2025

lore
Share & tools
lore
#0 lore raw

cover Build once, test everywhere

From: Chuck Lever <[email protected]>

With apologies to the Java folks.

I've been looking for ways to make our workflow runs more efficient because my lab resources are limited, and moving to the cloud costs money per CPU second.

One thing that seems like an obvious win would be to build the test kernel one time (for example, after a merge/pull request), then re-use that kernel binary for all the workflows we want to run on it.

For libvirt, the usual way to do this is use the 9p option. There are a few cases that are important to me where that option does not work:

1. The control host is running late-model Fedora, and the test runners want to run old LTS kernels
2. The control host or test runners run a RHEL-based kernel, which does not support 9p, last I checked
3. Cloud, where there is no way for cloud-based test runners to mount the control host with 9p

That leaves me with building the test kernel in each test runner. That's a lot of kernel builds; and in the cloud, the kernel source code has to be cloned onto each runner from kernel.org, because we don't have repo mirroring there.

Ted pointed out another use case where build-once is great: when you want to under-provision the test runners to exercise code paths that handle resource exhaustion. Building the kernel on such test runners takes forever.

Cloud providers throw another curve: there is typically a per- tenant or per-availability zone limit on the number of CPU cores you can provision at once. So a workflow that starts several instances has to limit each instance to only one or two cores. Building the kernel on those instances also takes forever.

I really want something more like KOTD, where test kernels are built once, packaged, and placed somewhere that the runners can find and install them. I'm not real interested in setting up a persistent yum repo and builder, though; I'd like kdevops to manage all that for me.

So I got out my whittling knife and built this proof of concept where kdevops brings up a kernel builder node that is tailored for a fast kernel build (eg, it is one guest with 16 vCPUs). The test kernel is packaged (.rpm or .deb) and fetched to the control host, then the builder node is destroyed.

I added a top-level "make" target that uploads the test kernel packages to each test runner, and they install it.

This is not part of the patch series, but shows how to run it:

$ make mrproper defconfig-kernel-builder
$ make && make bringup && make linux-packages && make destroy
$ make defconfig-workflow-one
$ make && make bringup && make linux-artifacts && make destroy
$ make defconfig-workflow-two
$ make && make bringup && make linux-artifacts && make destroy
$ make defconfig-workflow-three
$ make && make bringup && make linux-artifacts && make destroy
$ make mrproper

All three workflows use the same kernel, built just once, for all of their test runners.

This is still band-aids and chewing gum, but it seems to work on both libvirt and cloud configurations. There is more than one way to skin this cat, though.

Chuck Lever (3):
Add a guest/instance for building the test kernel
playbooks: Add a build_linux role
Experimental: Add a separate install_linux role

 .gitignore                                    |   2 +
 playbooks/build_linux.yml                     |   4 +
 playbooks/install_linux.yml                   |   4 +
 playbooks/roles/build_linux/README.md         |  74 +++++
 playbooks/roles/build_linux/defaults/main.yml |  38 +++
 .../tasks/install-deps/debian/main.yml        |  46 +++
 .../tasks/install-deps/redhat/main.yml        | 102 ++++++
 .../tasks/install-deps/suse/main.yml          |  31 ++
 playbooks/roles/build_linux/tasks/main.yml    | 295 ++++++++++++++++++
 playbooks/roles/gen_hosts/defaults/main.yml   |   2 +
 playbooks/roles/gen_hosts/tasks/main.yml      |  12 +
 .../roles/gen_hosts/templates/builder.j2      |  13 +
 playbooks/roles/gen_nodes/defaults/main.yml   |   2 +
 playbooks/roles/gen_nodes/tasks/main.yml      |  22 ++
 playbooks/roles/install_linux/README.md       | 136 ++++++++
 .../roles/install_linux/defaults/main.yml     |  43 +++
 .../tasks/install-deps/debian/main.yml        |  44 +++
 .../tasks/install-deps/redhat/main.yml        |  76 +++++
 .../tasks/install-deps/suse/main.yml          |  31 ++
 playbooks/roles/install_linux/tasks/main.yml  | 142 +++++++++
 .../tasks/update-grub/debian.yml              |   8 +
 .../tasks/update-grub/install.yml             | 196 ++++++++++++
 .../install_linux/tasks/update-grub/main.yml  |  15 +
 .../tasks/update-grub/redhat.yml              |  36 +++
 .../install_linux/tasks/update-grub/suse.yml  |  11 +
 workflows/linux/Kconfig                       |  10 +
 workflows/linux/Makefile                      |  19 ++
 27 files changed, 1414 insertions(+)
 create mode 100644 playbooks/build_linux.yml
 create mode 100644 playbooks/install_linux.yml
 create mode 100644 playbooks/roles/build_linux/README.md
 create mode 100644 playbooks/roles/build_linux/defaults/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/install-deps/debian/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/install-deps/redhat/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/install-deps/suse/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/main.yml
 create mode 100644 playbooks/roles/gen_hosts/templates/builder.j2
 create mode 100644 playbooks/roles/install_linux/README.md
 create mode 100644 playbooks/roles/install_linux/defaults/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/install-deps/debian/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/install-deps/redhat/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/install-deps/suse/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/debian.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/install.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/redhat.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/suse.yml
Signature · 1 line
2.49.0
#1 lore raw

RFC PATCH 1/3 Add a guest/instance for building the test kernel

From: Chuck Lever <[email protected]>

Provision a separate guest/instance for the purpose of building the Linux kernel under test. It's not used yet.

---
 playbooks/roles/gen_hosts/defaults/main.yml   |  2 ++
 playbooks/roles/gen_hosts/tasks/main.yml      | 12 ++++++++++
 .../roles/gen_hosts/templates/builder.j2      | 13 +++++++++++
 playbooks/roles/gen_nodes/defaults/main.yml   |  2 ++
 playbooks/roles/gen_nodes/tasks/main.yml      | 22 +++++++++++++++++++
 workflows/linux/Kconfig                       | 10 +++++++++
 workflows/linux/Makefile                      |  4 ++++
 7 files changed, 65 insertions(+)
 create mode 100644 playbooks/roles/gen_hosts/templates/builder.j2
playbooks/roles/gen_hosts/defaults/main.yml +2 -0
index f6ab9bcce166..85afb35040e3 100644
@@ -29,6 +29,8 @@ kdevops_workflow_enable_ltp: False
 kdevops_workflow_enable_nfstest: false
 kdevops_workflow_enable_sysbench: false
 
+bootlinux_builder: false
+
 is_fstests: False
 fstests_fstyp: "bogus"
 fs_config_role_path: "/dev/null"
playbooks/roles/gen_hosts/tasks/main.yml +12 -0
index a50355f72160..9182ac46f8c1 100644
@@ -56,6 +56,18 @@
   when:
     - is_fstests
 
+- name: Generate the Ansible hosts file for a Linux kernel build
+  tags: [ 'hosts' ]
+  template:
+    src: "{{ kdevops_hosts_template }}"
+    dest: "{{ topdir_path }}/{{ kdevops_hosts }}"
+    force: yes
+    trim_blocks: True
+    lstrip_blocks: True
+  when:
+    - bootlinux_builder
+    - ansible_hosts_template.stat.exists
+
 - name: Generate the Ansible hosts file
   tags: [ 'hosts' ]
   template:
playbooks/roles/gen_hosts/templates/builder.j2 new file +13 -0
new file mode 100644
index 000000000000..0c9ba1e8e01a
@@ -0,0 +1,13 @@
+[all]
+{{ kdevops_hosts_prefix }}-builder
+[all:vars]
+ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
+
+[baseline]
+{{ kdevops_hosts_prefix }}-builder
+[baseline:vars]
+ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
+
+[dev]
+[dev:vars]
+ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
playbooks/roles/gen_nodes/defaults/main.yml +2 -0
index 8ff9b87993a7..d5fa444c995f 100644
@@ -93,6 +93,8 @@ fstests_fstyp: "bogus"
 fs_config_role_path: "/dev/null"
 fs_config_data: "[section_1]"
 
+bootlinux_builder: false
+builder_nodes: []
 bootlinux_9p: False
 bootlinux_9p_host_path: "/dev/null"
 bootlinux_9p_msize: 0
playbooks/roles/gen_nodes/tasks/main.yml +22 -0
index d541dcbf1f54..e3d0127d1f65 100644
@@ -43,6 +43,14 @@
   when:
     - kdevops_baseline_and_dev
 
+- name: Set builder nodes array
+  tags: vars
+  set_fact:
+    builder_nodes:
+      - "{{ kdevops_host_prefix + '-builder' }}"
+  when:
+    - bootlinux_builder
+
 - name: Set iscsi_nodes list
   ansible.builtin.set_fact:
     iscsi_nodes: "{{ [kdevops_host_prefix + '-iscsi'] }}"
@@ -138,6 +146,20 @@
     - not kdevops_workflows_dedicated_workflow
     - ansible_nodes_template.stat.exists
 
+- name: Generate the builder kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template
+  tags: [ 'nodes' ]
+  vars:
+    node_template: "{{ kdevops_nodes_template | basename }}"
+    all_generic_nodes: "{{ builder_nodes }}"
+    nodes: "{{ all_generic_nodes }}"
+  template:
+    src: "{{ node_template }}"
+    dest: "{{ topdir_path }}/{{ kdevops_nodes }}"
+    force: yes
+  when:
+    - bootlinux_builder
+    - ansible_nodes_template.stat.exists
+
 - name: Generate the pynfs kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template
   tags: [ 'nodes' ]
   vars:
workflows/linux/Kconfig +10 -0
index 797469e60d20..8a26eb2e2897 100644
@@ -36,6 +36,16 @@ config BOOTLINUX_PURE_IOMAP
 
 endif # HAVE_SUPPORTS_PURE_IOMAP
 
+config BOOTLINUX_BUILDER
+	bool "Build once, test many"
+	output yaml
+	default n
+	help
+	  Enabling this option creates a separate guest/instance
+	  where the Linux kernel is built. kdevops passes the
+	  artifacts of this build to the test runner nodes to be
+	  installed before each workflow runs.
+
 config BOOTLINUX_9P
 	bool "Use 9p to build Linux"
 	depends on LIBVIRT && !GUESTFS_LACKS_9P
workflows/linux/Makefile +4 -0
index ecce273a4f67..65bbb8ae9a90 100644
@@ -13,6 +13,10 @@ ifeq (y,$(CONFIG_BOOTLINUX_PURE_IOMAP))
 TREE_CONFIG:=config-$(TREE_REF)-pure-iomap
 endif
 
+ifeq (y,$(CONFIG_BOOTLINUX_BUILDER))
+KDEVOPS_HOSTS_TEMPLATE=builder.j2
+endif
+
 # Describes the Linux clone
 BOOTLINUX_ARGS	+= target_linux_git=$(TREE_URL)
 # ifeq (y,$(CONFIG_BOOTLINUX_TREE_CUSTOM_NAME))
Signature · 1 line
2.49.0
#2 lore raw

RFC PATCH 2/3 playbooks: Add a build_linux role

In reply to [email protected]

From: Chuck Lever <[email protected]>

Currently, for cloud configurations or when 9p is disabled, kdevops builds the test kernel on each test runner. This is inefficient, and gets worse as the test matrix for a single kernel version scales out.

Instead we want to build the test kernel once and make the build artifacts available for test runners to install. This would be similar to what KOTD does now, except it does not require setting up a separate yum repo.

For this experiment, I've created a stripped down version of the bootlinux role that has only the steps needed to build the kernel on a target node. It adds some steps to create kernel packages and then fetch them to the control host.

---
 .gitignore                                    |   2 +
 playbooks/build_linux.yml                     |   4 +
 playbooks/roles/build_linux/README.md         |  74 +++++
 playbooks/roles/build_linux/defaults/main.yml |  38 +++
 .../tasks/install-deps/debian/main.yml        |  46 +++
 .../tasks/install-deps/redhat/main.yml        | 102 ++++++
 .../tasks/install-deps/suse/main.yml          |  31 ++
 playbooks/roles/build_linux/tasks/main.yml    | 295 ++++++++++++++++++
 workflows/linux/Makefile                      |   7 +
 9 files changed, 599 insertions(+)
 create mode 100644 playbooks/build_linux.yml
 create mode 100644 playbooks/roles/build_linux/README.md
 create mode 100644 playbooks/roles/build_linux/defaults/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/install-deps/debian/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/install-deps/redhat/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/install-deps/suse/main.yml
 create mode 100644 playbooks/roles/build_linux/tasks/main.yml
.gitignore +2 -0
index 45113a669390..f51213a59ad9 100644
@@ -86,6 +86,8 @@ playbooks/roles/linux-mirror/linux-mirror-systemd/mirrors.yaml
 #   yet.
 workflows/selftests/results/
 
+workflows/linux/artifacts/
+
 workflows/linux/refs/default/Kconfig.linus
 workflows/linux/refs/default/Kconfig.next
 workflows/linux/refs/default/Kconfig.stable
playbooks/build_linux.yml new file +4 -0
new file mode 100644
index 000000000000..094441d1befd
@@ -0,0 +1,4 @@
+---
+- hosts: all
+  roles:
+    - role: build_linux
playbooks/roles/build_linux/README.md new file +74 -0

74 lines not on this page · raw message

playbooks/roles/build_linux/defaults/main.yml new file +38 -0
new file mode 100644
index 000000000000..5ff5ece28b1e
@@ -0,0 +1,38 @@
+# SPDX-License-Identifier copyleft-next-0.3.1
+---
+kdevops_bootlinux: false
+infer_uid_and_group: false
+
+data_path: "/data"
+data_user: "kdevops"
+data_group: "kdevops"
+data_device: "/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops0"
+data_fstype: "xfs"
+data_label: "data"
+data_fs_opts: "-L {{ disk_setup_label }}"
+
+target_linux_admin_name: "Hacker Amanda"
+target_linux_admin_email: "[email protected]"
+target_linux_git: "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git"
+target_linux_shallow_depth: 0
+target_linux_tree: "linux-stable"
+target_linux_dir_path: "{{ data_path }}/{{ target_linux_tree }}"
+kdevops_baseline_and_dev: false
+
+target_linux_ref: "v4.19.133"
+target_linux_delta_file:
+target_linux_config: "config-{{ target_linux_ref }}"
+make: "make"
+target_linux_make_cmd: "{{ make }} -j{{ ansible_processor_nproc }}"
+target_linux_make_install_cmd: "{{ target_linux_make_cmd }} modules_install install"
+
+build_artifacts_dir: "{{ topdir_path }}/workflows/linux/artifacts/"
+
+uninstall_kernel_enable: false
+
+bootlinux_b4_am_this_host: false
+
+kdevops_workflow_enable_cxl: false
+bootlinux_cxl_test: false
+
+bootlinux_tree_set_by_cli: false
playbooks/roles/build_linux/tasks/install-deps/debian/main.yml new file +46 -0

46 lines not on this page · raw message

playbooks/roles/build_linux/tasks/install-deps/redhat/main.yml new file +102 -0

102 lines not on this page · raw message

playbooks/roles/build_linux/tasks/install-deps/suse/main.yml new file +31 -0
new file mode 100644
index 000000000000..0260da5a912d
@@ -0,0 +1,31 @@
+---
+- name: Install Linux kernel build dependencies for SUSE
+  become: true
+  become_method: ansible.builtin.sudo
+  community.general.zypper:
+    name:
+      - bison
+      - flex
+      - git-core
+      - gcc
+      - make
+      - gawk
+      - bc
+      - dump
+      - sed
+      - libopenssl-devel
+      - libelf-devel
+      - liburcu8
+      - diffutils
+      - net-tools
+      - ncurses-devel
+      - xfsprogs
+      - e2fsprogs
+      - btrfsprogs
+      - ntfs-3g
+      - mdadm
+      - rpcbind
+      - portmap
+      - hwinfo
+      - open-iscsi
+    disable_recommends: false
playbooks/roles/build_linux/tasks/main.yml new file +295 -0

295 lines not on this page · raw message

workflows/linux/Makefile +7 -0
index 65bbb8ae9a90..bb7441e71fda 100644
@@ -84,6 +84,7 @@ linux-help-menu:
 	@echo "linux-clone        - Only clones Linux"
 	@echo "linux-grub-setup   - Ensures the appropriate target kernel is set to boot"
 	@echo "linux-reboot       - Reboot guests"
+	@echo "linux-packages     - Clones, builds, and packages a Linux kernel"
 	@echo "uname              - Prints current running kernel"
 
 PHONY += linux-help-end
@@ -159,6 +160,12 @@ linux-reboot:
 		$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/bootlinux.yml \
 		--extra-vars="$(BOOTLINUX_ARGS)" $(LIMIT_HOSTS) --tags vars,reboot
 
+PHONY += linux-packages
+linux-packages:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -i $(KDEVOPS_HOSTFILE) \
+		$(KDEVOPS_PLAYBOOKS_DIR)/build_linux.yml \
+		--extra-vars="$(BOOTLINUX_ARGS)" $(LIMIT_HOSTS)
+
 PHONY += uname
 uname:
 	$(Q)ansible all -i hosts -b -m command -a "uname -r" -o \
Signature · 1 line
2.49.0
#3 lore raw

RFC PATCH 3/3 Experimental: Add a separate install_linux role

In reply to [email protected]

From: Chuck Lever <[email protected]>

Add a role that can grab the artifacts in workflows/linux/artifacts and install them on all guests/instances.

---
 playbooks/install_linux.yml                   |   4 +
 playbooks/roles/install_linux/README.md       | 136 ++++++++++++
 .../roles/install_linux/defaults/main.yml     |  43 ++++
 .../tasks/install-deps/debian/main.yml        |  44 ++++
 .../tasks/install-deps/redhat/main.yml        |  76 +++++++
 .../tasks/install-deps/suse/main.yml          |  31 +++
 playbooks/roles/install_linux/tasks/main.yml  | 142 +++++++++++++
 .../tasks/update-grub/debian.yml              |   8 +
 .../tasks/update-grub/install.yml             | 196 ++++++++++++++++++
 .../install_linux/tasks/update-grub/main.yml  |  15 ++
 .../tasks/update-grub/redhat.yml              |  36 ++++
 .../install_linux/tasks/update-grub/suse.yml  |  11 +
 workflows/linux/Makefile                      |   8 +
 13 files changed, 750 insertions(+)
 create mode 100644 playbooks/install_linux.yml
 create mode 100644 playbooks/roles/install_linux/README.md
 create mode 100644 playbooks/roles/install_linux/defaults/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/install-deps/debian/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/install-deps/redhat/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/install-deps/suse/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/debian.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/install.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/main.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/redhat.yml
 create mode 100644 playbooks/roles/install_linux/tasks/update-grub/suse.yml
playbooks/install_linux.yml new file +4 -0
new file mode 100644
index 000000000000..273ef4512f81
@@ -0,0 +1,4 @@
+---
+- hosts: all
+  roles:
+    - role: install_linux
playbooks/roles/install_linux/README.md new file +136 -0

136 lines not on this page · raw message

playbooks/roles/install_linux/defaults/main.yml new file +43 -0

43 lines not on this page · raw message

playbooks/roles/install_linux/tasks/install-deps/debian/main.yml new file +44 -0

44 lines not on this page · raw message

playbooks/roles/install_linux/tasks/install-deps/redhat/main.yml new file +76 -0

76 lines not on this page · raw message

playbooks/roles/install_linux/tasks/install-deps/suse/main.yml new file +31 -0
new file mode 100644
index 000000000000..204a181b8237
@@ -0,0 +1,31 @@
+---
+- name: Install Linux kernel build dependencies for SUSE sources
+  become: yes
+  become_method: sudo
+  zypper:
+    name:
+      - bison
+      - flex
+      - git-core
+      - gcc
+      - make
+      - gawk
+      - bc
+      - dump
+      - sed
+      - libopenssl-devel
+      - libelf-devel
+      - liburcu8
+      - diffutils
+      - net-tools
+      - ncurses-devel
+      - xfsprogs
+      - e2fsprogs
+      - btrfsprogs
+      - ntfs-3g
+      - mdadm
+      - rpcbind
+      - portmap
+      - hwinfo
+      - open-iscsi
+    disable_recommends: no
playbooks/roles/install_linux/tasks/main.yml new file +142 -0

142 lines not on this page · raw message

playbooks/roles/install_linux/tasks/update-grub/debian.yml new file +8 -0
new file mode 100644
index 000000000000..3c7deea2161a
@@ -0,0 +1,8 @@
+- name: Run update-grub
+  become: yes
+  become_flags: 'su - -c'
+  become_method: sudo
+  command: "update-grub"
+  register: grub_update
+  changed_when: "grub_update.rc == 0"
+  tags: [ 'linux', 'manual-update-grub', 'console' ]
playbooks/roles/install_linux/tasks/update-grub/install.yml new file +196 -0

196 lines not on this page · raw message

playbooks/roles/install_linux/tasks/update-grub/main.yml new file +15 -0
new file mode 100644
index 000000000000..a565e0ac26ac
@@ -0,0 +1,15 @@
+---
+- name: Debian-specific grub update
+  ansible.builtin.import_tasks: debian.yml
+  when:
+    - ansible_os_family == "Debian"
+
+- name: Red Hat-specific grub update
+  ansible.builtin.import_tasks: redhat.yml
+  when:
+    - ansible_os_family == "RedHat"
+
+- name: Suse-specific grub update
+  ansible.builtin.import_tasks: suse.yml
+  when:
+    - ansible_os_family == "Suse"
playbooks/roles/install_linux/tasks/update-grub/redhat.yml new file +36 -0
new file mode 100644
index 000000000000..11a92f34bab6
@@ -0,0 +1,36 @@
+- name: Disable Grub menu auto-hide
+  become: true
+  become_flags: 'su - -c'
+  become_method: ansible.builtin.sudo
+  ansible.builtin.command: "grub2-editenv - unset menu_auto_hide"
+  register: grub_edit
+  changed_when: "grub_edit.rc == 0"
+
+- name: Determine if system was booted using UEFI
+  ansible.builtin.stat:
+    path: "/sys/firmware/efi/efivars"
+  register: efi_boot
+
+- name: Use /etc/grub2.cfg as the grub configuration file
+  ansible.builtin.set_fact:
+    grub_config_file: "/etc/grub2.cfg"
+  when:
+    - not efi_boot.stat.exists
+
+- name: Use /etc/grub2-efi.cfg as the configuration file
+  ansible.builtin.set_fact:
+    grub_config_file: "/etc/grub2-efi.cfg"
+  when:
+    - efi_boot.stat.exists
+
+- name: Run update-grub
+  become: true
+  become_flags: 'su - -c'
+  become_method: ansible.builtin.sudo
+  ansible.builtin.command: "grub2-mkconfig -o {{ grub_config_file }}"
+  register: grub_update
+  changed_when: "grub_update.rc == 0"
+  tags:
+    - linux
+    - manual-update-grub
+    - console
playbooks/roles/install_linux/tasks/update-grub/suse.yml new file +11 -0
new file mode 100644
index 000000000000..1cb1fdc55f58
@@ -0,0 +1,11 @@
+- name: Run update-grub
+  become: true
+  become_flags: 'su - -c'
+  become_method: ansible.builtin.sudo
+  ansible.builtin.command: "update-bootloader --refresh"
+  register: grub_update
+  changed_when: "grub_update.rc == 0"
+  tags:
+    - linux
+    - manual-update-grub
+    - console
workflows/linux/Makefile +8 -0
index bb7441e71fda..06722d5903b3 100644
@@ -85,6 +85,7 @@ linux-help-menu:
 	@echo "linux-grub-setup   - Ensures the appropriate target kernel is set to boot"
 	@echo "linux-reboot       - Reboot guests"
 	@echo "linux-packages     - Clones, builds, and packages a Linux kernel"
+	@echo "linux-artifacts    - Installs artifacts generated by 'linux-packages'"
 	@echo "uname              - Prints current running kernel"
 
 PHONY += linux-help-end
@@ -166,6 +167,13 @@ linux-packages:
 		$(KDEVOPS_PLAYBOOKS_DIR)/build_linux.yml \
 		--extra-vars="$(BOOTLINUX_ARGS)" $(LIMIT_HOSTS)
 
+PHONY += linux-artifacts
+linux-artifacts:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+		-i $(KDEVOPS_HOSTFILE) \
+		$(KDEVOPS_PLAYBOOKS_DIR)/install_linux.yml \
+		--extra-vars="$(BOOTLINUX_ARGS)" $(LIMIT_HOSTS)
+
 PHONY += uname
 uname:
 	$(Q)ansible all -i hosts -b -m command -a "uname -r" -o \
Signature · 1 line
2.49.0
#4 lore raw
In reply to [email protected]
Quoted 21 lines · On Tue, Apr 22, 2025 at 11:49:03AM -0400, [email protected] wrote:

So I got out my whittling knife and built this proof of concept where kdevops brings up a kernel builder node that is tailored for a fast kernel build (eg, it is one guest with 16 vCPUs). The test kernel is packaged (.rpm or .deb) and fetched to the control host, then the builder node is destroyed.

I added a top-level "make" target that uploads the test kernel packages to each test runner, and they install it.

This is not part of the patch series, but shows how to run it:

$ make mrproper defconfig-kernel-builder
$ make && make bringup && make linux-packages && make destroy
$ make defconfig-workflow-one
$ make && make bringup && make linux-artifacts && make destroy
$ make defconfig-workflow-two
$ make && make bringup && make linux-artifacts && make destroy
$ make defconfig-workflow-three
$ make && make bringup && make linux-artifacts && make destroy
$ make mrproper
Quoted 6 lines

All three workflows use the same kernel, built just once, for all of their test runners.

This is still band-aids and chewing gum, but it seems to work on both libvirt and cloud configurations. There is more than one way to skin this cat, though.

This all looks good to me. We also have a slew of CIs which could take advantage of daily linux and linux-next builds, when that is a target kernel, perhaps we could leverage opensuse build service for that as that I think supports letting us leverage rpm and deb binaries.

But yeah all this work in this series makes perfect sense to me!

Luis

#5 lore raw
In reply to [email protected]
Quoted 4 lines · On Tue, Apr 22, 2025 at 11:49:03AM +0100, [email protected] wrote:

From: Chuck Lever <[email protected]>

With apologies to the Java folks.

Quoted 8 lines

I've been looking for ways to make our workflow runs more efficient because my lab resources are limited, and moving to the cloud costs money per CPU second.

One thing that seems like an obvious win would be to build the test kernel one time (for example, after a merge/pull request), then re-use that kernel binary for all the workflows we want to run on it.

What about rebuilds between iterations?
I think an extra step to improve that build time could be adding support for ccache in our kernel builds. Would it make sense to extend this in the future to enable a redis node?

https://github.com/ccache/ccache/wiki/Redis-storage

With the workflow suggestion below, I'd then do:

1. Bring up a redis node guest
2. Bring up a kernel builder node that pulls from redis node ccache
3. Push redis artifacts to control host (for later sync up) and destroy redis guest
4. Push kernel package to control host

That said, all this new workflow makes sense to me. I guess instead of duplicating the bootlinux role into build_linux I'd like to see a linux role that handles build location (localhost/specific guest), output package generation or not, deployment, etc. Does that make sense to you too?

#6 lore raw
Quoted 19 lines · Daniel Gomez
Quoted 4 lines · On Tue, Apr 22, 2025 at 11:49:03AM +0100, [email protected] wrote:

From: Chuck Lever <[email protected]>

With apologies to the Java folks.

Quoted 8 lines

I've been looking for ways to make our workflow runs more efficient because my lab resources are limited, and moving to the cloud costs money per CPU second.

One thing that seems like an obvious win would be to build the test kernel one time (for example, after a merge/pull request), then re-use that kernel binary for all the workflows we want to run on it.

What about rebuilds between iterations?
I think an extra step to improve that build time could be adding support for ccache in our kernel builds. Would it make sense to extend this in the future to enable a redis node?

Quoted 8 lines

https://github.com/ccache/ccache/wiki/Redis-storage

With the workflow suggestion below, I'd then do:

1. Bring up a redis node guest
2. Bring up a kernel builder node that pulls from redis node ccache
3. Push redis artifacts to control host (for later sync up) and destroy redis guest
4. Push kernel package to control host

So I've asked our internal testing folks about using ccache to speed up kernel builds. There were various reasons not to do this, one of which is we want to have a clean trustworthy and repeatable kernel build to test with.

I'm not sure I have a specific technical argument against using ccache except that our builder node is ephemeral, making its ccache usable for only a single build. A Redis storage node might solve that issue. However I wonder if the time it takes to bring up the node, reload its storage, then after the build save the redis artifacts and destroy it might wipe out any benefit.

With libvirt guests we can save several minutes by doing the guest bring up in parallel (via Ansible) instead of serially via a shell script. I've got some ideas about that.

Quoted 4 lines

That said, all this new workflow makes sense to me. I guess instead of duplicating the bootlinux role into build_linux I'd like to see a linux role that handles build location (localhost/specific guest), output package generation or not, deployment, etc. Does that make sense to you too?

The bootlinux role has grown a slew of special cases over the years, so I'm in the mood to help reorganize it a little.

At least split it into a role that handles the kernel build, and one that handles the minutiae of installing the builds on test runners.

I'm not quite following your suggestion, but I'm interested in hearing more detail.

Signature · 1 line
Chuck Lever
#7 lore raw
Quoted 29 lines · Chuck Lever
Quoted 19 lines · Daniel Gomez
Quoted 4 lines · On Tue, Apr 22, 2025 at 11:49:03AM +0100, [email protected] wrote:

From: Chuck Lever <[email protected]>

With apologies to the Java folks.

Quoted 8 lines

I've been looking for ways to make our workflow runs more efficient because my lab resources are limited, and moving to the cloud costs money per CPU second.

One thing that seems like an obvious win would be to build the test kernel one time (for example, after a merge/pull request), then re-use that kernel binary for all the workflows we want to run on it.

What about rebuilds between iterations?
I think an extra step to improve that build time could be adding support for ccache in our kernel builds. Would it make sense to extend this in the future to enable a redis node?

Quoted 8 lines

https://github.com/ccache/ccache/wiki/Redis-storage

With the workflow suggestion below, I'd then do:

1. Bring up a redis node guest
2. Bring up a kernel builder node that pulls from redis node ccache
3. Push redis artifacts to control host (for later sync up) and destroy redis guest
4. Push kernel package to control host

Quoted 8 lines

So I've asked our internal testing folks about using ccache to speed up kernel builds. There were various reasons not to do this, one of which is we want to have a clean trustworthy and repeatable kernel build to test with.

I'm not sure I have a specific technical argument against using ccache except that our builder node is ephemeral, making its ccache usable for only a single build. A Redis storage node might solve that issue.

Build artifacts can also be shared between the build nodes instances as long as they are the same following reproducible builds [1]

[1]
https://docs.kernel.org/kbuild/reproducible-builds.html#reproducible-builds

But all right, I understand this is a separate topic and having ccache/redis/ reproducible build features is something we can work with and extend as opt-in features in kdevops and/or this new workflow.

Thanks for the feedback.

Quoted 3 lines

However I wonder if the time it takes to bring up the node, reload its storage, then after the build save the redis artifacts and destroy it might wipe out any benefit.

I guess the benefits will depend on every user, kernel config and how the cache is exposed.

Quoted 6 lines

With libvirt guests we can save several minutes by doing the guest bring up in parallel (via Ansible) instead of serially via a shell script. I've got some ideas about that.

Quoted 7 lines
Quoted 4 lines

That said, all this new workflow makes sense to me. I guess instead of duplicating the bootlinux role into build_linux I'd like to see a linux role that handles build location (localhost/specific guest), output package generation or not, deployment, etc. Does that make sense to you too?

The bootlinux role has grown a slew of special cases over the years, so I'm in the mood to help reorganize it a little.

That would be great.

Quoted 6 lines

At least split it into a role that handles the kernel build, and one that handles the minutiae of installing the builds on test runners.

I'm not quite following your suggestion, but I'm interested in hearing more detail.

I'm usually more in favor of having standalone playbooks. But I see here the potential of merging the features rather than having them separate. My suggestion is to refactor bootlinux role into separate tasks/<task name/ feature>. Where the features are build_linux, artifacts, package creation, etc.

Quoted 4 lines
Signature · 2 lines
-- 
Chuck Lever
#8 lore raw
Quoted 78 lines · Daniel Gomez
Quoted 29 lines · Chuck Lever
Quoted 19 lines · Daniel Gomez
Quoted 4 lines · On Tue, Apr 22, 2025 at 11:49:03AM +0100, [email protected] wrote:

From: Chuck Lever <[email protected]>

With apologies to the Java folks.

Quoted 8 lines

I've been looking for ways to make our workflow runs more efficient because my lab resources are limited, and moving to the cloud costs money per CPU second.

One thing that seems like an obvious win would be to build the test kernel one time (for example, after a merge/pull request), then re-use that kernel binary for all the workflows we want to run on it.

What about rebuilds between iterations?
I think an extra step to improve that build time could be adding support for ccache in our kernel builds. Would it make sense to extend this in the future to enable a redis node?

Quoted 8 lines

https://github.com/ccache/ccache/wiki/Redis-storage

With the workflow suggestion below, I'd then do:

1. Bring up a redis node guest
2. Bring up a kernel builder node that pulls from redis node ccache
3. Push redis artifacts to control host (for later sync up) and destroy redis guest
4. Push kernel package to control host

Quoted 8 lines

So I've asked our internal testing folks about using ccache to speed up kernel builds. There were various reasons not to do this, one of which is we want to have a clean trustworthy and repeatable kernel build to test with.

I'm not sure I have a specific technical argument against using ccache except that our builder node is ephemeral, making its ccache usable for only a single build. A Redis storage node might solve that issue.

Build artifacts can also be shared between the build nodes instances as long as they are the same following reproducible builds [1]

[1]
https://docs.kernel.org/kbuild/reproducible-builds.html#reproducible-builds

But all right, I understand this is a separate topic and having ccache/redis/ reproducible build features is something we can work with and extend as opt-in features in kdevops and/or this new workflow.

Thanks for the feedback.

Quoted 3 lines

However I wonder if the time it takes to bring up the node, reload its storage, then after the build save the redis artifacts and destroy it might wipe out any benefit.

I guess the benefits will depend on every user, kernel config and how the cache is exposed.

Quoted 6 lines

With libvirt guests we can save several minutes by doing the guest bring up in parallel (via Ansible) instead of serially via a shell script. I've got some ideas about that.

Quoted 7 lines
Quoted 4 lines

That said, all this new workflow makes sense to me. I guess instead of duplicating the bootlinux role into build_linux I'd like to see a linux role that handles build location (localhost/specific guest), output package generation or not, deployment, etc. Does that make sense to you too?

The bootlinux role has grown a slew of special cases over the years, so I'm in the mood to help reorganize it a little.

That would be great.

Quoted 4 lines

At least split it into a role that handles the kernel build, and one that handles the minutiae of installing the builds on test runners.

Quoted 7 lines
Quoted 2 lines

I'm not quite following your suggestion, but I'm interested in hearing more detail.

I'm usually more in favor of having standalone playbooks. But I see here the potential of merging the features rather than having them separate. My suggestion is to refactor bootlinux role into separate tasks/<task name/ feature>. Where the features are build_linux, artifacts, package creation, etc.

I did something similar to playbooks/roles/gen_nodes. I can experiment and post some ideas.

Signature · 1 line
Chuck Lever