Skip to contentSkip to conversation

kdevops: add initial nixos support

PATCH In review Luis Chamberlain 1 participant 5 messages Aug 27, 2025 – Aug 29, 2025

lore
Share & tools
lore
#0 lore raw

cover kdevops: add initial nixos support

This adds a barebones nixos support. Enough to get ssh going and initial mirror support. The mirror works but its empty as it has no packages. So all we can do is bringup. Enough to kick the tires off. Some initial packages are added in preparation for future workflows suppor but that is a bit more complex and requires moer work. This should be enough to let the ball get rolling.

Luis Chamberlain (3):
common: use fallback for group inference on remote systems
nixos: add NixOS support as third bringup option with libvirt
integration
mirror: add Nix binary cache mirroring support

 .gitignore                                    |   3 +
 defconfigs/mirror                             |   1 +
 defconfigs/nixos                              |  27 +
 docs/kdevops-nixos.md                         | 404 +++++++++++++
 kconfigs/Kconfig.bringup                      |  22 +-
 kconfigs/Kconfig.mirror                       |  68 ++-
 kconfigs/Kconfig.nixos                        | 130 +++++
 nixos/flake.nix                               |  32 ++
 .../files/scripts/detect_libvirt_session.sh   |  26 +
 playbooks/linux-mirror.yml                    |   2 +
 playbooks/nixos.yml                           | 540 ++++++++++++++++++
 playbooks/roles/common/tasks/main.yml         |  29 +-
 .../devconfig/tasks/install-deps/main.yml     |   1 +
 playbooks/roles/devconfig/tasks/main.yml      |   4 +-
 playbooks/roles/gen_hosts/tasks/main.yml      |  15 +
 .../roles/gen_hosts/templates/fstests.j2      |  20 +
 playbooks/roles/gen_hosts/templates/hosts.j2  |  16 +
 playbooks/roles/gen_nodes/tasks/main.yml      |  24 +
 .../roles/gen_nodes/templates/nixos_nodes.j2  |  14 +
 .../linux-mirror/python/start-mirroring.py    |  29 +-
 .../roles/nix-cache-mirror/defaults/main.yml  |   7 +
 .../roles/nix-cache-mirror/handlers/main.yml  |  11 +
 .../roles/nix-cache-mirror/tasks/main.yml     | 161 ++++++
 .../templates/nix-cache-mirror-cache.conf.j2  |   3 +
 .../templates/nix-cache-mirror.nginx.j2       |  81 +++
 .../templates/nix-cache-sync.service.j2       |  17 +
 .../templates/nix-cache-sync.timer.j2         |  11 +
 .../roles/update_etc_hosts/tasks/main.yml     |   2 +
 .../templates/nixos/configuration.nix.j2      | 138 +++++
 playbooks/templates/nixos/flake.nix.j2        |  38 ++
 .../nixos/hardware-configuration.nix.j2       |  42 ++
 .../templates/nixos/run-vm-wrapper.sh.j2      | 159 ++++++
 playbooks/templates/nixos/vm-libvirt.xml.j2   |  96 ++++
 playbooks/templates/nixos/vms.nix.j2          |  45 ++
 .../templates/nixos/workflow-deps.nix.j2      | 127 ++++
 playbooks/update_ssh_config_nixos.yml         |  57 ++
 scripts/check_nix_mirror.sh                   |  66 +++
 scripts/detect_libvirt_session.sh             |  26 +
 scripts/nixos.Makefile                        |  93 +++
 scripts/nixos_ssh_key_name.py                 |  55 ++
 scripts/provision.Makefile                    |   4 +
 scripts/status_nixos.sh                       |  57 ++
 scripts/update_ssh_config_nixos.py            | 133 +++++
 43 files changed, 2826 insertions(+), 10 deletions(-)
 create mode 100644 defconfigs/nixos
 create mode 100644 docs/kdevops-nixos.md
 create mode 100644 kconfigs/Kconfig.nixos
 create mode 100644 nixos/flake.nix
 create mode 100755 playbooks/files/scripts/detect_libvirt_session.sh
 create mode 100644 playbooks/nixos.yml
 create mode 100644 playbooks/roles/gen_nodes/templates/nixos_nodes.j2
 create mode 100644 playbooks/roles/nix-cache-mirror/defaults/main.yml
 create mode 100644 playbooks/roles/nix-cache-mirror/handlers/main.yml
 create mode 100644 playbooks/roles/nix-cache-mirror/tasks/main.yml
 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2
 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2
 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2
 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2
 create mode 100644 playbooks/templates/nixos/configuration.nix.j2
 create mode 100644 playbooks/templates/nixos/flake.nix.j2
 create mode 100644 playbooks/templates/nixos/hardware-configuration.nix.j2
 create mode 100644 playbooks/templates/nixos/run-vm-wrapper.sh.j2
 create mode 100644 playbooks/templates/nixos/vm-libvirt.xml.j2
 create mode 100644 playbooks/templates/nixos/vms.nix.j2
 create mode 100644 playbooks/templates/nixos/workflow-deps.nix.j2
 create mode 100644 playbooks/update_ssh_config_nixos.yml
 create mode 100755 scripts/check_nix_mirror.sh
 create mode 100755 scripts/detect_libvirt_session.sh
 create mode 100644 scripts/nixos.Makefile
 create mode 100755 scripts/nixos_ssh_key_name.py
 create mode 100755 scripts/status_nixos.sh
 create mode 100755 scripts/update_ssh_config_nixos.py
Signature · 1 line
2.50.1
#1 lore raw

PATCH 1/3 common: use fallback for group inference on remote systems

When provisioning remote bare metal systems or VMs with different user configurations, getent group <username> may fail if the user's primary group has a different name. This is common in:
- Enterprise environments with LDAP/AD integration
- Bare metal systems with pre-existing user configurations
- Systems like NixOS with different group naming conventions
- Cloud/managed hosting with auto-generated group names

Add fallback to extract the primary GID from the passwd entry and look up the group by GID, which is more reliable across diverse system configurations.

---
 playbooks/roles/common/tasks/main.yml | 29 +++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)
playbooks/roles/common/tasks/main.yml +27 -2
index 3ee08018..bb92fea8 100644
@@ -55,9 +55,34 @@
   when:
     - infer_uid_and_group|bool
 
-- name: Override user and group with inferred settings if feature is enabled
-  ansible.builtin.set_fact:
+- name: Get primary group ID from user entry
+  set_fact:
+    user_primary_gid: "{{ getent_running_user.ansible_facts.getent_passwd[target_user][2] }}"
+  when:
+    - infer_uid_and_group|bool
+    - getent_on_group.rc | default(1) != 0
+
+- name: Run getent against the primary group ID
+  getent:
+    database: group
+    key: "{{ user_primary_gid }}"
+  register: getent_primary_group
+  when:
+    - infer_uid_and_group|bool
+    - getent_on_group.rc | default(1) != 0
+
+- name: Override user and group with inferred settings if feature is enabled (group found by name)
+  set_fact:
     data_user: "{{ target_user }}"
     data_group: "{{ ((getent_on_group.values() | first).values() | first).keys() | first }}"
   when:
     - infer_uid_and_group|bool
+    - getent_on_group.rc | default(1) == 0
+
+- name: Override user and group with inferred settings if feature is enabled (group found by GID)
+  set_fact:
+    data_user: "{{ target_user }}"
+    data_group: "{{ getent_primary_group.ansible_facts.getent_group.keys() | first }}"
+  when:
+    - infer_uid_and_group|bool
+    - getent_on_group.rc | default(1) != 0
Signature · 1 line
2.50.1
#2 lore raw

PATCH 2/3 nixos: add NixOS support as third bringup option with libvirt integration

In reply to Luis Chamberlain

This commit adds NixOS as a third bringup option alongside guestfs and SKIP_BRINGUP, providing a declarative and reproducible way to provision test VMs using NixOS's functional package management system with full libvirt integration.

Libvirt Integration:
- NixOS VMs are managed through libvirt system session for professional VM lifecycle
- Uses standard libvirt networking with DHCP assignment (192.168.122.x range)
- No port forwarding needed - direct SSH access to VM IP addresses
- Standard virsh commands for VM management (start, shutdown, destroy, console)
- Integrates with existing libvirt infrastructure and monitoring tools

SSH Session Management:
- SSH keys are dynamically generated based on directory location using
format: ~/.ssh/kdevops-nixos-<directory>-<hash>
- This ensures unique keys per kdevops instance, preventing conflicts between multiple deployments
- SSH config entries are automatically managed during bringup
- Direct connections to DHCP-assigned IPs (no port forwarding complexity)

Path Compatibility:
- NixOS uses different system paths than traditional Linux distributions
- Python interpreter: /run/current-system/sw/bin/python3 (not /usr/bin/python3)
- Bash shell: /run/current-system/sw/bin/bash (not /bin/bash)
- Templates automatically detect and use correct paths for NixOS

VM Management:
- Professional VM lifecycle through libvirt system session
- XML-based VM configuration with proper resource allocation
- QCOW2 disk images with virtio drivers for performance
- Automatic network configuration via libvirt's default network
- Full integration with existing libvirt monitoring and management tools

Workflow Support:
The following workflows have initial NixOS support with only package dependency resolution, actual run time tests will be needed later:
- fstests: Filesystem testing (XFS, Btrfs, EXT4)
- blktests: Block layer testing (NVMe, SCSI, NBD)
- selftests: Linux kernel selftests
- mmtests: Memory management performance testing
- sysbench: Database performance benchmarking
- pynfs: NFS protocol testing
- ltp: Linux Test Project
- gitr: Git regression testing

Key Features:
- Declarative configuration through Nix expressions
- Reproducible builds using Nix flakes
- Automatic dependency resolution for workflows
- Directory-based isolation for multiple kdevops instances
- Full libvirt integration for professional VM management
- Automatic SSH configuration during bringup
- Standard networking without port forwarding complexity

This implementation provides a modern, functional approach to VM provisioning that leverages both NixOS's strengths in reproducibility and libvirt's professional VM management capabilities.

Generated-by Claude AI
---
 .gitignore                                    |   3 +
 defconfigs/nixos                              |  27 +
 docs/kdevops-nixos.md                         | 404 ++++++++++++++
 kconfigs/Kconfig.bringup                      |  22 +-
 kconfigs/Kconfig.nixos                        | 101 ++++
 nixos/flake.nix                               |  32 ++
 .../files/scripts/detect_libvirt_session.sh   |  26 +
 playbooks/nixos.yml                           | 516 ++++++++++++++++++
 .../devconfig/tasks/install-deps/main.yml     |   1 +
 playbooks/roles/devconfig/tasks/main.yml      |   4 +-
 playbooks/roles/gen_hosts/tasks/main.yml      |  15 +
 .../roles/gen_hosts/templates/fstests.j2      |  20 +
 playbooks/roles/gen_hosts/templates/hosts.j2  |  16 +
 playbooks/roles/gen_nodes/tasks/main.yml      |  24 +
 .../roles/gen_nodes/templates/nixos_nodes.j2  |  14 +
 .../roles/update_etc_hosts/tasks/main.yml     |   2 +
 .../templates/nixos/configuration.nix.j2      | 138 +++++
 playbooks/templates/nixos/flake.nix.j2        |  38 ++
 .../nixos/hardware-configuration.nix.j2       |  42 ++
 .../templates/nixos/run-vm-wrapper.sh.j2      | 159 ++++++
 playbooks/templates/nixos/vm-libvirt.xml.j2   |  96 ++++
 playbooks/templates/nixos/vms.nix.j2          |  45 ++
 .../templates/nixos/workflow-deps.nix.j2      | 127 +++++
 playbooks/update_ssh_config_nixos.yml         |  57 ++
 scripts/detect_libvirt_session.sh             |  26 +
 scripts/nixos.Makefile                        |  93 ++++
 scripts/nixos_ssh_key_name.py                 |  55 ++
 scripts/provision.Makefile                    |   4 +
 scripts/status_nixos.sh                       |  57 ++
 scripts/update_ssh_config_nixos.py            | 133 +++++
 30 files changed, 2294 insertions(+), 3 deletions(-)
 create mode 100644 defconfigs/nixos
 create mode 100644 docs/kdevops-nixos.md
 create mode 100644 kconfigs/Kconfig.nixos
 create mode 100644 nixos/flake.nix
 create mode 100755 playbooks/files/scripts/detect_libvirt_session.sh
 create mode 100644 playbooks/nixos.yml
 create mode 100644 playbooks/roles/gen_nodes/templates/nixos_nodes.j2
 create mode 100644 playbooks/templates/nixos/configuration.nix.j2
 create mode 100644 playbooks/templates/nixos/flake.nix.j2
 create mode 100644 playbooks/templates/nixos/hardware-configuration.nix.j2
 create mode 100644 playbooks/templates/nixos/run-vm-wrapper.sh.j2
 create mode 100644 playbooks/templates/nixos/vm-libvirt.xml.j2
 create mode 100644 playbooks/templates/nixos/vms.nix.j2
 create mode 100644 playbooks/templates/nixos/workflow-deps.nix.j2
 create mode 100644 playbooks/update_ssh_config_nixos.yml
 create mode 100755 scripts/detect_libvirt_session.sh
 create mode 100644 scripts/nixos.Makefile
 create mode 100755 scripts/nixos_ssh_key_name.py
 create mode 100755 scripts/status_nixos.sh
 create mode 100755 scripts/update_ssh_config_nixos.py
.gitignore +3 -0
index 75e4712d..2bea9d48 100644
@@ -102,3 +102,6 @@ scripts/kconfig/.nconf-cfg
 Kconfig.passthrough_libvirt.generated
 
 archive/
+
+# NixOS generated files
+nixos/generated/
defconfigs/nixos new file +27 -0
new file mode 100644
index 00000000..c510e341
@@ -0,0 +1,27 @@
+CONFIG_NIXOS=y
+CONFIG_LIBVIRT=y
+
+# Disable mirror setup for NixOS
+CONFIG_ENABLE_LOCAL_LINUX_MIRROR=n
+CONFIG_USE_LOCAL_LINUX_MIRROR=n
+CONFIG_INSTALL_LOCAL_LINUX_MIRROR=n
+CONFIG_MIRROR_INSTALL=n
+
+CONFIG_NIXOS_USE_FLAKES=y
+CONFIG_NIXOS_CHANNEL="nixos-unstable"
+CONFIG_NIXOS_ENABLE_WORKFLOW_DEPS=y
+CONFIG_NIXOS_LIBVIRT_SESSION_INFERENCE=y
+
+CONFIG_NIXOS_VM_MEMORY_MB=4096
+CONFIG_NIXOS_VM_DISK_SIZE_GB=20
+CONFIG_NIXOS_VM_VCPUS=4
+
+CONFIG_WORKFLOWS=y
+CONFIG_WORKFLOW_LINUX_CUSTOM=y
+
+CONFIG_BOOTLINUX=y
+CONFIG_BOOTLINUX_9P=n
+
+CONFIG_KDEVOPS_TRY_REFRESH_REPOS=y
+CONFIG_KDEVOPS_TRY_UPDATE_SYSTEMS=y
+CONFIG_KDEVOPS_TRY_INSTALL_KDEV_TOOLS=y
docs/kdevops-nixos.md new file +404 -0

404 lines not on this page · raw message

kconfigs/Kconfig.bringup +21 -1
index 887d3851..8caf07be 100644
@@ -5,6 +5,10 @@ config KDEVOPS_ENABLE_GUESTFS
 	bool
 	output yaml
 
+config KDEVOPS_ENABLE_NIXOS
+	bool
+	output yaml
+
 choice
 	prompt "Node bring up method"
 	default GUESTFS
@@ -39,6 +43,21 @@ config TERRAFORM
 
 	  If you are not using a cloud environment just disable this.
 
+config NIXOS
+	bool "NixOS declarative configuration with libvirt"
+	select KDEVOPS_ENABLE_NIXOS
+	select EXTRA_STORAGE_SUPPORTS_512
+	select EXTRA_STORAGE_SUPPORTS_1K
+	select EXTRA_STORAGE_SUPPORTS_2K
+	select EXTRA_STORAGE_SUPPORTS_4K
+	select EXTRA_STORAGE_SUPPORTS_LARGEIO
+	help
+	  Use NixOS declarative configuration system to provision VMs with
+	  libvirt. This provides a purely functional approach to VM management
+	  with automatic dependency resolution based on enabled workflows.
+	  NixOS will automatically infer the libvirt session type (system vs
+	  user) based on your distribution's defaults, similar to guestfs.
+
 config SKIP_BRINGUP
 	bool "Skip bring up - bare metal or existing nodes"
 	select EXTRA_STORAGE_SUPPORTS_512
@@ -55,10 +74,11 @@ endchoice
 
 config LIBVIRT
 	bool
-	depends on GUESTFS
+	depends on GUESTFS || NIXOS
 	default y
 
 source "kconfigs/Kconfig.guestfs"
+source "kconfigs/Kconfig.nixos"
 source "terraform/Kconfig"
 if LIBVIRT
 source "kconfigs/Kconfig.libvirt"
kconfigs/Kconfig.nixos new file +101 -0

101 lines not on this page · raw message

nixos/flake.nix new file +32 -0
new file mode 100644
index 00000000..530d5980
@@ -0,0 +1,32 @@
+{
+  description = "kdevops NixOS VMs";
+
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+  };
+
+  outputs = { self, nixpkgs }: {
+    nixosConfigurations = {
+      "kdevops" = nixpkgs.lib.nixosSystem {
+        system = "x86_64-linux";
+        modules = [
+          ./generated/configuration.nix
+          ./generated/hardware-configuration.nix
+          ./generated/workflow-deps.nix
+          ({ ... }: {
+            networking.hostName = "kdevops";
+          })
+        ];
+      };
+    };
+
+    # Build all VMs
+    defaultPackage.x86_64-linux =
+      nixpkgs.legacyPackages.x86_64-linux.writeShellScriptBin "build-vms" ''
+        echo "Building NixOS VMs..."
+        echo "Building kdevops..."
+        nix build .#nixosConfigurations.kdevops.config.system.build.vm
+        echo "All VMs built successfully!"
+      '';
+  };
+}
playbooks/files/scripts/detect_libvirt_session.sh new file +26 -0
new file mode 100755
index 00000000..caea9367
@@ -0,0 +1,26 @@
+#!/bin/bash
+# SPDX-License-Identifier: copyleft-next-0.3.1
+#
+# Detect the appropriate libvirt session type (system vs user) based on
+# distribution defaults, similar to how guestfs handles it.
+
+SCRIPTS_DIR=$(dirname $0)
+source ${SCRIPTS_DIR}/libvirt_pool.sh
+
+OS_FILE="/etc/os-release"
+LIBVIRT_URI="qemu:///system"  # Default to system
+
+# Get the pool variables which includes distribution detection
+get_pool_vars
+
+# Fedora defaults to user session
+if [[ "$USES_QEMU_USER_SESSION" == "y" ]]; then
+    LIBVIRT_URI="qemu:///session"
+fi
+
+# Override detection if explicitly configured
+if [[ -n "$CONFIG_LIBVIRT_URI_PATH" ]]; then
+    LIBVIRT_URI="$CONFIG_LIBVIRT_URI_PATH"
+fi
+
+echo "$LIBVIRT_URI"
playbooks/nixos.yml new file +516 -0

516 lines not on this page · raw message

playbooks/roles/devconfig/tasks/install-deps/main.yml +1 -0
index 68ad9e7b..3cca4d9b 100644
@@ -22,6 +22,7 @@
     - files:
         - "{{ ansible_facts['os_family'] | lower }}.yml"
       skip: true
+  when: ansible_facts['os_family'] != 'NixOS'
   tags: vars
 
 - name: Debian-specific setup
playbooks/roles/devconfig/tasks/main.yml +2 -2
index 2ffa433f..fccd1fcf 100644
@@ -197,7 +197,7 @@
       chmod 755 {{ dev_bash_config }}
     fi
   args:
-    executable: /bin/bash
+    executable: "{{ '/run/current-system/sw/bin/bash' if (kdevops_enable_nixos | default(false)) else '/bin/bash' }}"
   when: dev_bash_config_file_copied is success
 
 - name: Copy the developer's favorite bash hacks over for root *if* it exists
@@ -224,7 +224,7 @@
       chmod 755 {{ dev_bash_config_root }}
     fi
   args:
-    executable: /bin/bash
+    executable: "{{ '/run/current-system/sw/bin/bash' if (kdevops_enable_nixos | default(false)) else '/bin/bash' }}"
   when: dev_bash_config_file_copied_root is success
 
 - name: Check to see if system has GRUB2
playbooks/roles/gen_hosts/tasks/main.yml +15 -0
index d36790b0..fb63629a 100644
@@ -79,6 +79,20 @@
   when:
     - not kdevops_workflows_dedicated_workflow
     - ansible_hosts_template.stat.exists
+    - not kdevops_enable_nixos|default(false)|bool
+
+- name: Generate the Ansible inventory file for NixOS
+  tags: ['hosts']
+  ansible.builtin.template:
+    src: "{{ kdevops_hosts_template }}"
+    dest: "{{ ansible_cfg_inventory }}"
+    force: true
+    trim_blocks: True
+    lstrip_blocks: True
+  when:
+    - not kdevops_workflows_dedicated_workflow
+    - ansible_hosts_template.stat.exists
+    - kdevops_enable_nixos|default(false)|bool
 
 - name: Update Ansible inventory access modification time so make sees it updated
   ansible.builtin.file:
@@ -339,6 +353,7 @@
     - kdevops_workflows_dedicated_workflow
     - kdevops_workflow_enable_fio_tests
     - ansible_hosts_template.stat.exists
+    - not kdevops_enable_nixos|default(false)|bool
 
 
 - name: Infer enabled mmtests test types
playbooks/roles/gen_hosts/templates/fstests.j2 +20 -0
index 32d90abf..823dbb1e 100644
@@ -1,10 +1,18 @@
 [all]
 localhost ansible_connection=local
 {% for s in fstests_enabled_test_types %}
+{% if kdevops_enable_nixos|default(false) %}
+{{ kdevops_host_prefix }}-{{ s }} ansible_python_interpreter=/run/current-system/sw/bin/python3
+{% else %}
 {{ kdevops_host_prefix }}-{{ s }}
+{% endif %}
 {% if kdevops_baseline_and_dev %}
+{% if kdevops_enable_nixos|default(false) %}
+{{ kdevops_host_prefix }}-{{ s }}-dev ansible_python_interpreter=/run/current-system/sw/bin/python3
+{% else %}
 {{ kdevops_host_prefix }}-{{ s }}-dev
 {% endif %}
+{% endif %}
 {% endfor %}
 {% if kdevops_nfsd_enable %}
 {% if kdevops_loopback_nfs_enable %}
@@ -15,7 +23,11 @@ localhost ansible_connection=local
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 [baseline]
 {% for s in fstests_enabled_test_types %}
+{% if kdevops_enable_nixos|default(false) %}
+{{ kdevops_host_prefix }}-{{ s }} ansible_python_interpreter=/run/current-system/sw/bin/python3
+{% else %}
 {{ kdevops_host_prefix }}-{{ s }}
+{% endif %}
 {% endfor %}
 {% if kdevops_nfsd_enable %}
 {% if kdevops_loopback_nfs_enable %}
@@ -27,7 +39,11 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 [dev]
 {% if kdevops_baseline_and_dev %}
   {% for s in fstests_enabled_test_types %}
+{% if kdevops_enable_nixos|default(false) %}
+{{ kdevops_host_prefix }}-{{ s }}-dev ansible_python_interpreter=/run/current-system/sw/bin/python3
+{% else %}
 {{ kdevops_host_prefix }}-{{ s }}-dev
+{% endif %}
   {% endfor %}
 {% if kdevops_nfsd_enable %}
 {% if kdevops_loopback_nfs_enable %}
@@ -62,7 +78,11 @@ ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 [krb5]
 {% for s in fstests_enabled_test_types %}
+{% if kdevops_enable_nixos|default(false) %}
+{{ kdevops_host_prefix }}-{{ s }} ansible_python_interpreter=/run/current-system/sw/bin/python3
+{% else %}
 {{ kdevops_host_prefix }}-{{ s }}
+{% endif %}
 {% endfor %}
 {% if kdevops_nfsd_enable %}
 {{ kdevops_hosts_prefix }}-nfsd
playbooks/roles/gen_hosts/templates/hosts.j2 +16 -0
index e9441605..0e896481 100644
@@ -184,10 +184,18 @@ write-your-own-template-for-your-workflow-and-task
 {% else %}
 [all]
 localhost ansible_connection=local
+{% if kdevops_enable_nixos|default(false) %}
+{{ kdevops_host_prefix }} ansible_python_interpreter=/run/current-system/sw/bin/python3
+{% else %}
 {{ kdevops_host_prefix }}
+{% endif %}
 {% if kdevops_baseline_and_dev == True %}
+{% if kdevops_enable_nixos|default(false) %}
+{{ kdevops_host_prefix }}-dev ansible_python_interpreter=/run/current-system/sw/bin/python3
+{% else %}
 {{ kdevops_host_prefix }}-dev
 {% endif %}
+{% endif %}
 {% if kdevops_enable_iscsi %}
 {{ kdevops_host_prefix }}-iscsi
 {% endif %}
@@ -197,13 +205,21 @@ localhost ansible_connection=local
 [all:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 [baseline]
+{% if kdevops_enable_nixos|default(false) %}
+{{ kdevops_host_prefix }} ansible_python_interpreter=/run/current-system/sw/bin/python3
+{% else %}
 {{ kdevops_host_prefix }}
+{% endif %}
 [baseline:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 [dev]
 {% if kdevops_baseline_and_dev %}
+{% if kdevops_enable_nixos|default(false) %}
+{{ kdevops_host_prefix }}-dev ansible_python_interpreter=/run/current-system/sw/bin/python3
+{% else %}
 {{ kdevops_host_prefix }}-dev
 {% endif %}
+{% endif %}
 [dev:vars]
 ansible_python_interpreter =  "{{ kdevops_python_interpreter }}"
 {% if kdevops_enable_iscsi %}
playbooks/roles/gen_nodes/tasks/main.yml +24 -0
index b294d294..b1a1946f 100644
@@ -27,6 +27,12 @@
     mode: "0755"
   when: kdevops_enable_guestfs|bool
 
+- name: Create nixos directory
+  ansible.builtin.file:
+    path: "{{ nixos_config_dir }}"
+    state: directory
+  when: kdevops_enable_nixos | default(false) | bool
+
 - name: Verify Ansible nodes template file exists {{ kdevops_nodes_template_full_path }}
   ansible.builtin.stat:
     path: "{{ kdevops_nodes_template_full_path }}"
@@ -148,6 +154,23 @@
     mode: "0644"
   when:
     - not kdevops_workflows_dedicated_workflow
+    - ansible_nodes_template.stat.exists
+    - not kdevops_enable_nixos|default(false)|bool
+
+- name: Generate the NixOS kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template
+  tags: ['nodes']
+  vars:
+    node_template: "{{ kdevops_nodes_template | basename }}"
+    all_generic_nodes: "{{ generic_nodes }}"
+    nodes: "{{ all_generic_nodes }}"
+  ansible.builtin.template:
+    src: "{{ node_template }}"
+    dest: "{{ topdir_path }}/{{ kdevops_nodes }}"
+    force: true
+  when:
+    - not kdevops_workflows_dedicated_workflow
+    - ansible_nodes_template.stat.exists
+    - kdevops_enable_nixos|default(false)|bool
 
 
 - name: Generate the builder kdevops nodes file using nodes file using template as jinja2 source template
@@ -162,6 +185,7 @@
     force: true
   when:
     - bootlinux_builder
+    - ansible_nodes_template.stat.exists
 
 
 - name: Generate the pynfs kdevops nodes file using nodes file using template as jinja2 source template
playbooks/roles/gen_nodes/templates/nixos_nodes.j2 new file +14 -0
new file mode 100644
index 00000000..391b1f10
@@ -0,0 +1,14 @@
+---
+# Ansible nodes file generated for NixOS VMs
+
+{% for node in nodes %}
+{{ node }}:
+  ansible_host: 192.168.100.{{ loop.index + 1 }}
+  ansible_user: kdevops
+  ansible_ssh_private_key_file: {{ topdir_path }}/.ssh/kdevops_id_rsa
+  ansible_python_interpreter: /run/current-system/sw/bin/python3
+  vm_name: {{ node }}
+  vm_memory_mb: {{ nixos_vm_memory_mb }}
+  vm_vcpus: {{ nixos_vm_vcpus }}
+  vm_disk_size_gb: {{ nixos_vm_disk_size_gb }}
+{% endfor %}
playbooks/roles/update_etc_hosts/tasks/main.yml +2 -0
index 049411ee..dc40eded 100644
@@ -57,6 +57,7 @@
   with_items: "{{ ueh_hosts }}"
   when:
     - terraform_private_net_enabled
+    - not (kdevops_enable_nixos | default(false))
 
 - name: Add IP address of all hosts to all hosts
   become: true
@@ -69,6 +70,7 @@
   with_items: "{{ ueh_hosts }}"
   when:
     - not terraform_private_net_enabled
+    - not (kdevops_enable_nixos | default(false))
 
 - name: Fix up hostname on Debian guestfs hosts
   become: true
playbooks/templates/nixos/configuration.nix.j2 new file +138 -0

138 lines not on this page · raw message

playbooks/templates/nixos/flake.nix.j2 new file +38 -0
new file mode 100644
index 00000000..52b5b680
@@ -0,0 +1,38 @@
+{
+  description = "kdevops NixOS VMs";
+
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/{{ nixos_channel }}";
+  };
+
+  outputs = { self, nixpkgs }: {
+    nixosConfigurations = {
+{% for node in groups['all'] if node != 'localhost' %}
+      "{{ node }}" = nixpkgs.lib.nixosSystem {
+        system = "x86_64-linux";
+        modules = [
+          ./generated/configuration.nix
+          ./generated/hardware-configuration.nix
+{% if nixos_enable_workflow_deps %}
+          ./generated/workflow-deps.nix
+{% endif %}
+          ({ ... }: {
+            networking.hostName = "{{ node }}";
+          })
+        ];
+      };
+{% endfor %}
+    };
+
+    # Build all VMs
+    defaultPackage.x86_64-linux =
+      nixpkgs.legacyPackages.x86_64-linux.writeShellScriptBin "build-vms" ''
+        echo "Building NixOS VMs..."
+{% for node in groups['all'] if node != 'localhost' %}
+        echo "Building {{ node }}..."
+        nix build .#nixosConfigurations.{{ node }}.config.system.build.vm
+{% endfor %}
+        echo "All VMs built successfully!"
+      '';
+  };
+}
playbooks/templates/nixos/hardware-configuration.nix.j2 new file +42 -0

42 lines not on this page · raw message

playbooks/templates/nixos/run-vm-wrapper.sh.j2 new file +159 -0

159 lines not on this page · raw message

playbooks/templates/nixos/vm-libvirt.xml.j2 new file +96 -0

96 lines not on this page · raw message

playbooks/templates/nixos/vms.nix.j2 new file +45 -0

45 lines not on this page · raw message

playbooks/templates/nixos/workflow-deps.nix.j2 new file +127 -0

127 lines not on this page · raw message

playbooks/update_ssh_config_nixos.yml new file +57 -0

57 lines not on this page · raw message

scripts/detect_libvirt_session.sh new file +26 -0
new file mode 100755
index 00000000..caea9367
@@ -0,0 +1,26 @@
+#!/bin/bash
+# SPDX-License-Identifier: copyleft-next-0.3.1
+#
+# Detect the appropriate libvirt session type (system vs user) based on
+# distribution defaults, similar to how guestfs handles it.
+
+SCRIPTS_DIR=$(dirname $0)
+source ${SCRIPTS_DIR}/libvirt_pool.sh
+
+OS_FILE="/etc/os-release"
+LIBVIRT_URI="qemu:///system"  # Default to system
+
+# Get the pool variables which includes distribution detection
+get_pool_vars
+
+# Fedora defaults to user session
+if [[ "$USES_QEMU_USER_SESSION" == "y" ]]; then
+    LIBVIRT_URI="qemu:///session"
+fi
+
+# Override detection if explicitly configured
+if [[ -n "$CONFIG_LIBVIRT_URI_PATH" ]]; then
+    LIBVIRT_URI="$CONFIG_LIBVIRT_URI_PATH"
+fi
+
+echo "$LIBVIRT_URI"
scripts/nixos.Makefile new file +93 -0

93 lines not on this page · raw message

scripts/nixos_ssh_key_name.py new file +55 -0

55 lines not on this page · raw message

scripts/provision.Makefile +4 -0
index a5ab8b84..f04264f6 100644
@@ -60,6 +60,10 @@ ifeq (y,$(CONFIG_GUESTFS))
 include scripts/guestfs.Makefile
 endif
 
+ifeq (y,$(CONFIG_NIXOS))
+include scripts/nixos.Makefile
+endif
+
 KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_SSH)
 KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_DEVCONFIG)
 
scripts/status_nixos.sh new file +57 -0

57 lines not on this page · raw message

scripts/update_ssh_config_nixos.py new file +133 -0

133 lines not on this page · raw message

Signature · 1 line
2.50.1
#3 lore raw

PATCH 3/3 mirror: add Nix binary cache mirroring support

In reply to Luis Chamberlain

Add support for a Nix binary cache mirroring infrastructure for enterprise environments to reduce latencies and public bandwidth consumption for CIs. This dramatically reduces NixOS package download times from hundreds of packages to local cache hits.

Key features:
- nginx-based proxy caching for Nix binary cache
- Automatic mirror detection for client-side configuration
- systemd service and timer for cache synchronization
- Integration with existing defconfig-mirror infrastructure
- Client auto-configuration when local mirrors are available

Components:
- playbooks/roles/nix-cache-mirror/: Complete Ansible role
- scripts/check_nix_mirror.sh: Mirror detection and URL discovery
- kconfigs/Kconfig.mirror: Mirror configuration options
- Integration with defconfig-mirror for one-time setup

The mirror setup follows existing kdevops mirror patterns and integrates with the git mirror infrastructure already in place.

Generated-by Claude AI
---
 defconfigs/mirror                             |   1 +
 kconfigs/Kconfig.mirror                       |  68 +++++++-
 kconfigs/Kconfig.nixos                        |  29 ++++
 playbooks/linux-mirror.yml                    |   2 +
 playbooks/nixos.yml                           |  28 ++-
 .../linux-mirror/python/start-mirroring.py    |  29 +++-
 .../roles/nix-cache-mirror/defaults/main.yml  |   7 +
 .../roles/nix-cache-mirror/handlers/main.yml  |  11 ++
 .../roles/nix-cache-mirror/tasks/main.yml     | 161 ++++++++++++++++++
 .../templates/nix-cache-mirror-cache.conf.j2  |   3 +
 .../templates/nix-cache-mirror.nginx.j2       |  81 +++++++++
 .../templates/nix-cache-sync.service.j2       |  17 ++
 .../templates/nix-cache-sync.timer.j2         |  11 ++
 scripts/check_nix_mirror.sh                   |  66 +++++++
 14 files changed, 507 insertions(+), 7 deletions(-)
 create mode 100644 playbooks/roles/nix-cache-mirror/defaults/main.yml
 create mode 100644 playbooks/roles/nix-cache-mirror/handlers/main.yml
 create mode 100644 playbooks/roles/nix-cache-mirror/tasks/main.yml
 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2
 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2
 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2
 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2
 create mode 100755 scripts/check_nix_mirror.sh
defconfigs/mirror +1 -0
index efa0f9ef..05d51c01 100644
@@ -2,3 +2,4 @@ CONFIG_SKIP_BRINGUP=y
 CONFIG_WORKFLOWS=n
 CONFIG_INSTALL_LOCAL_LINUX_MIRROR=y
 CONFIG_LINUX_MIRROR_NFS=y
+CONFIG_INSTALL_NIX_CACHE_MIRROR=y
kconfigs/Kconfig.mirror +66 -2

85 lines not on this page · raw message

kconfigs/Kconfig.nixos +29 -0
index 55361215..3def5372 100644
@@ -91,6 +91,35 @@ config NIXOS_SSH_PORT
 	help
 	  SSH port to use for connecting to NixOS VMs.
 
+config NIXOS_USE_LOCAL_MIRROR
+	bool "Use local Nix binary cache mirror"
+	output yaml
+	default $(shell, scripts/check_nix_mirror.sh USE_NIX_CACHE_MIRROR)
+	help
+	  Enable this to use a local Nix binary cache mirror if available.
+	  This dramatically speeds up NixOS builds by avoiding repeated
+	  downloads of packages from cache.nixos.org.
+
+	  When enabled, kdevops will automatically detect and configure
+	  VMs to use the local mirror if available at /mirror/nix-cache
+	  or via HTTP on port 8080.
+
+	  This requires that you have set up a mirror server using the
+	  defconfig-mirror configuration.
+
+config NIXOS_MIRROR_URL
+	string "Nix binary cache mirror URL"
+	output yaml
+	default $(shell, scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL)
+	depends on NIXOS_USE_LOCAL_MIRROR
+	help
+	  The URL of the local Nix binary cache mirror. This is
+	  automatically detected based on available mirrors:
+	  - HTTP mirror if running on port 8080
+	  - File-based mirror at /mirror/nix-cache (for NFS mounts)
+
+	  Leave empty to auto-detect at runtime.
+
 config NIXOS_DEBUG_MODE
 	bool "Enable debug mode for NixOS provisioning"
 	default n
playbooks/linux-mirror.yml +2 -0
index 2af0f89d..2ec39d31 100644
@@ -3,3 +3,5 @@
   hosts: localhost
   roles:
     - role: linux-mirror
+    - role: nix-cache-mirror
+      when: install_nix_cache_mirror | default(false) | bool
playbooks/nixos.yml +26 -2
index eda34586..bdc9b1e8 100644
@@ -95,6 +95,21 @@
           ansible.builtin.set_fact:
             nixos_ssh_authorized_key: "{{ ssh_public_key['content'] | b64decode | trim }}"
 
+    - name: Detect local Nix cache mirror URL if enabled
+      ansible.builtin.shell: |
+        bash {{ playbook_dir }}/../scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL
+      register: detected_mirror_url
+      when: nixos_use_local_mirror | default(false) | bool and (nixos_mirror_url is not defined or nixos_mirror_url == "")
+      changed_when: false
+
+    - name: Set detected mirror URL
+      ansible.builtin.set_fact:
+        nixos_mirror_url: "{{ detected_mirror_url.stdout | trim }}"
+      when:
+        - detected_mirror_url is defined
+        - detected_mirror_url.stdout is defined
+        - detected_mirror_url.stdout | trim != ""
+
     - name: Template base NixOS configuration
       ansible.builtin.template:
         src: nixos/configuration.nix.j2
@@ -176,9 +191,18 @@
         fi
 
         # Configure Nix to use local mirror if available
-        {% if nixos_use_local_mirror is defined and nixos_use_local_mirror and nixos_mirror_url is defined and nixos_mirror_url != "" %}
+        {% if nixos_use_local_mirror is defined and nixos_use_local_mirror %}
+        # Dynamically detect mirror URL if not provided
+        {% if nixos_mirror_url is not defined or nixos_mirror_url == "" %}
+        DETECTED_MIRROR=$(bash {{ playbook_dir }}/../scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL)
+        if [ -n "$DETECTED_MIRROR" ]; then
+          export NIX_CONFIG="substituters = $DETECTED_MIRROR https://cache.nixos.org"
+          echo "Using detected local Nix cache mirror: $DETECTED_MIRROR"
+        fi
+        {% else %}
         export NIX_CONFIG="substituters = {{ nixos_mirror_url }} https://cache.nixos.org"
-        echo "Using local Nix cache mirror: {{ nixos_mirror_url }}"
+        echo "Using configured local Nix cache mirror: {{ nixos_mirror_url }}"
+        {% endif %}
         {% endif %}
 
         cd {{ nixos_generation_dir }}
playbooks/roles/linux-mirror/python/start-mirroring.py +26 -3
index 03ede449..c7c9d81e 100755
@@ -54,7 +54,8 @@ def mirror_entry(mirror, args):
     cmd = cmd + reference_args
     mirror_target = mirror_path + target
     if os.path.isdir(mirror_target):
-        return
+        sys.stdout.write("Skipping %s - mirror already exists at %s\n" % (short_name, mirror_target))
+        return "skipped"
     sys.stdout.write("Mirroring: %s onto %s\n" % (short_name, mirror_target))
     if args.verbose:
         sys.stdout.write("%s\n" % (cmd))
@@ -74,6 +75,7 @@ def mirror_entry(mirror, args):
         process.wait()
         if process.returncode != 0:
             raise Exception(f"Failed clone with:\n%s" % (" ".join(cmd)))
+        return "cloned"
 
 
 def main():
@@ -134,15 +136,36 @@ def main():
                 % (mirror.get("short_name"), args.yaml_mirror, total)
             )
 
+    # Statistics tracking
+    stats = {"skipped": 0, "cloned": 0, "failed": 0}
     # Mirror trees without a reference first
     for mirror in yaml_vars["mirrors"]:
         if not mirror.get("reference"):
-            mirror_entry(mirror, args)
+            result = mirror_entry(mirror, args)
+            if result == "skipped":
+                stats["skipped"] += 1
+            elif result == "cloned":
+                stats["cloned"] += 1
+            else:
+                stats["failed"] += 1
 
     # Mirror trees which need a reference last
     for mirror in yaml_vars["mirrors"]:
         if mirror.get("reference"):
-            mirror_entry(mirror, args)
+            result = mirror_entry(mirror, args)
+            if result == "skipped":
+                stats["skipped"] += 1
+            elif result == "cloned":
+                stats["cloned"] += 1
+            else:
+                stats["failed"] += 1
+    # Print summary
+    sys.stdout.write("\n=== Mirror Summary ===\n")
+    sys.stdout.write("Total repositories: %d\n" % total)
+    sys.stdout.write("Skipped (already exist): %d\n" % stats["skipped"])
+    sys.stdout.write("Cloned (new): %d\n" % stats["cloned"])
+    if stats["failed"] > 0:
+        sys.stdout.write("Failed: %d\n" % stats["failed"])
 
 
 if __name__ == "__main__":
playbooks/roles/nix-cache-mirror/defaults/main.yml new file +7 -0
new file mode 100644
index 00000000..02349b78
@@ -0,0 +1,7 @@
+---
+# Default variables for nix-cache-mirror role
+nix_cache_mirror_path: "/mirror/nix-cache"
+nix_cache_mirror_port: 8080
+nix_cache_upstream_url: "https://cache.nixos.org"
+nix_cache_mirror_nginx_conf_path: "/etc/nginx/sites-available/nix-cache-mirror"
+nix_cache_mirror_nginx_enabled_path: "/etc/nginx/sites-enabled/nix-cache-mirror"
playbooks/roles/nix-cache-mirror/handlers/main.yml new file +11 -0
new file mode 100644
index 00000000..d51ca036
@@ -0,0 +1,11 @@
+---
+- name: reload nginx
+  become: true
+  ansible.builtin.systemd:
+    name: nginx
+    state: reloaded
+
+- name: reload systemd
+  become: true
+  ansible.builtin.systemd:
+    daemon_reload: true
playbooks/roles/nix-cache-mirror/tasks/main.yml new file +161 -0

161 lines not on this page · raw message

playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2 new file +3 -0
new file mode 100644
index 00000000..77b0cda8
@@ -0,0 +1,3 @@
+# Nix cache mirror proxy cache configuration
+# This file must be included at http context level
+proxy_cache_path {{ nix_cache_mirror_path }}/nginx-cache levels=1:2 keys_zone=nixcache:100m max_size=50g inactive=7d use_temp_path=off;
playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2 new file +81 -0

81 lines not on this page · raw message

playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2 new file +17 -0
new file mode 100644
index 00000000..62aee320
@@ -0,0 +1,17 @@
+[Unit]
+Description=Nix Binary Cache Sync Service
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=www-data
+Group=www-data
+ExecStart=/usr/bin/curl -f -s "{{ mirror_nix_cache_upstream_url | default('https://cache.nixos.org') }}/nix-cache-info" -o "{{ nix_cache_mirror_path }}/nix-cache-info"
+ExecStartPost=/bin/bash -c 'find {{ nix_cache_mirror_path }} -name "*.narinfo" -mtime +7 -delete'
+ExecStartPost=/bin/bash -c 'find {{ nix_cache_mirror_path }} -name "*.nar*" -mtime +7 -delete'
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2 new file +11 -0
new file mode 100644
index 00000000..ef80ad36
@@ -0,0 +1,11 @@
+[Unit]
+Description=Run Nix Binary Cache Sync Service hourly
+Requires=nix-cache-sync.service
+
+[Timer]
+OnCalendar=hourly
+Persistent=true
+RandomizedDelaySec=300
+
+[Install]
+WantedBy=timers.target
scripts/check_nix_mirror.sh new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.50.1
#4 lore raw
In reply to Luis Chamberlain
Quoted 7 lines · Luis Chamberlain

This adds a barebones nixos support. Enough to get ssh going and initial mirror support. The mirror works but its empty as it has no packages. So all we can do is bringup. Enough to kick the tires off. Some initial packages are added in preparation for future workflows suppor but that is a bit more complex and requires moer work. This should be enough to let the ball get rolling.

Quoted 5 lines

Luis Chamberlain (3):
common: use fallback for group inference on remote systems
nixos: add NixOS support as third bringup option with libvirt
integration
mirror: add Nix binary cache mirroring support

Pushed.

Luis