kdevops: add initial nixos support
cover kdevops: add initial nixos support
This adds a barebones nixos support. Enough to get ssh going and initial mirror support. The mirror works but its empty as it has no packages. So all we can do is bringup. Enough to kick the tires off. Some initial packages are added in preparation for future workflows suppor but that is a bit more complex and requires moer work. This should be enough to let the ball get rolling.
Luis Chamberlain (3):
common: use fallback for group inference on remote systems
nixos: add NixOS support as third bringup option with libvirt
integration
mirror: add Nix binary cache mirroring support
.gitignore | 3 + defconfigs/mirror | 1 + defconfigs/nixos | 27 + docs/kdevops-nixos.md | 404 +++++++++++++ kconfigs/Kconfig.bringup | 22 +- kconfigs/Kconfig.mirror | 68 ++- kconfigs/Kconfig.nixos | 130 +++++ nixos/flake.nix | 32 ++ .../files/scripts/detect_libvirt_session.sh | 26 + playbooks/linux-mirror.yml | 2 + playbooks/nixos.yml | 540 ++++++++++++++++++ playbooks/roles/common/tasks/main.yml | 29 +- .../devconfig/tasks/install-deps/main.yml | 1 + playbooks/roles/devconfig/tasks/main.yml | 4 +- playbooks/roles/gen_hosts/tasks/main.yml | 15 + .../roles/gen_hosts/templates/fstests.j2 | 20 + playbooks/roles/gen_hosts/templates/hosts.j2 | 16 + playbooks/roles/gen_nodes/tasks/main.yml | 24 + .../roles/gen_nodes/templates/nixos_nodes.j2 | 14 + .../linux-mirror/python/start-mirroring.py | 29 +- .../roles/nix-cache-mirror/defaults/main.yml | 7 + .../roles/nix-cache-mirror/handlers/main.yml | 11 + .../roles/nix-cache-mirror/tasks/main.yml | 161 ++++++ .../templates/nix-cache-mirror-cache.conf.j2 | 3 + .../templates/nix-cache-mirror.nginx.j2 | 81 +++ .../templates/nix-cache-sync.service.j2 | 17 + .../templates/nix-cache-sync.timer.j2 | 11 + .../roles/update_etc_hosts/tasks/main.yml | 2 + .../templates/nixos/configuration.nix.j2 | 138 +++++ playbooks/templates/nixos/flake.nix.j2 | 38 ++ .../nixos/hardware-configuration.nix.j2 | 42 ++ .../templates/nixos/run-vm-wrapper.sh.j2 | 159 ++++++ playbooks/templates/nixos/vm-libvirt.xml.j2 | 96 ++++ playbooks/templates/nixos/vms.nix.j2 | 45 ++ .../templates/nixos/workflow-deps.nix.j2 | 127 ++++ playbooks/update_ssh_config_nixos.yml | 57 ++ scripts/check_nix_mirror.sh | 66 +++ scripts/detect_libvirt_session.sh | 26 + scripts/nixos.Makefile | 93 +++ scripts/nixos_ssh_key_name.py | 55 ++ scripts/provision.Makefile | 4 + scripts/status_nixos.sh | 57 ++ scripts/update_ssh_config_nixos.py | 133 +++++ 43 files changed, 2826 insertions(+), 10 deletions(-) create mode 100644 defconfigs/nixos create mode 100644 docs/kdevops-nixos.md create mode 100644 kconfigs/Kconfig.nixos create mode 100644 nixos/flake.nix create mode 100755 playbooks/files/scripts/detect_libvirt_session.sh create mode 100644 playbooks/nixos.yml create mode 100644 playbooks/roles/gen_nodes/templates/nixos_nodes.j2 create mode 100644 playbooks/roles/nix-cache-mirror/defaults/main.yml create mode 100644 playbooks/roles/nix-cache-mirror/handlers/main.yml create mode 100644 playbooks/roles/nix-cache-mirror/tasks/main.yml create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2 create mode 100644 playbooks/templates/nixos/configuration.nix.j2 create mode 100644 playbooks/templates/nixos/flake.nix.j2 create mode 100644 playbooks/templates/nixos/hardware-configuration.nix.j2 create mode 100644 playbooks/templates/nixos/run-vm-wrapper.sh.j2 create mode 100644 playbooks/templates/nixos/vm-libvirt.xml.j2 create mode 100644 playbooks/templates/nixos/vms.nix.j2 create mode 100644 playbooks/templates/nixos/workflow-deps.nix.j2 create mode 100644 playbooks/update_ssh_config_nixos.yml create mode 100755 scripts/check_nix_mirror.sh create mode 100755 scripts/detect_libvirt_session.sh create mode 100644 scripts/nixos.Makefile create mode 100755 scripts/nixos_ssh_key_name.py create mode 100755 scripts/status_nixos.sh create mode 100755 scripts/update_ssh_config_nixos.py
Signature · 1 line
2.50.1
PATCH 1/3 common: use fallback for group inference on remote systems
When provisioning remote bare metal systems or VMs with different user configurations, getent group <username> may fail if the user's primary group has a different name. This is common in:
- Enterprise environments with LDAP/AD integration
- Bare metal systems with pre-existing user configurations
- Systems like NixOS with different group naming conventions
- Cloud/managed hosting with auto-generated group names
Add fallback to extract the primary GID from the passwd entry and look up the group by GID, which is more reliable across diverse system configurations.
--- playbooks/roles/common/tasks/main.yml | 29 +++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-)
playbooks/roles/common/tasks/main.yml +27 -2
@@ -55,9 +55,34 @@ when: - infer_uid_and_group|bool -- name: Override user and group with inferred settings if feature is enabled - ansible.builtin.set_fact: +- name: Get primary group ID from user entry + set_fact: + user_primary_gid: "{{ getent_running_user.ansible_facts.getent_passwd[target_user][2] }}" + when: + - infer_uid_and_group|bool + - getent_on_group.rc | default(1) != 0 + +- name: Run getent against the primary group ID + getent: + database: group + key: "{{ user_primary_gid }}" + register: getent_primary_group + when: + - infer_uid_and_group|bool + - getent_on_group.rc | default(1) != 0 + +- name: Override user and group with inferred settings if feature is enabled (group found by name) + set_fact: data_user: "{{ target_user }}" data_group: "{{ ((getent_on_group.values() | first).values() | first).keys() | first }}" when: - infer_uid_and_group|bool + - getent_on_group.rc | default(1) == 0 + +- name: Override user and group with inferred settings if feature is enabled (group found by GID) + set_fact: + data_user: "{{ target_user }}" + data_group: "{{ getent_primary_group.ansible_facts.getent_group.keys() | first }}" + when: + - infer_uid_and_group|bool + - getent_on_group.rc | default(1) != 0
Signature · 1 line
2.50.1
PATCH 2/3 nixos: add NixOS support as third bringup option with libvirt integration
In reply to Luis ChamberlainThis commit adds NixOS as a third bringup option alongside guestfs and SKIP_BRINGUP, providing a declarative and reproducible way to provision test VMs using NixOS's functional package management system with full libvirt integration.
Libvirt Integration:
- NixOS VMs are managed through libvirt system session for professional VM lifecycle
- Uses standard libvirt networking with DHCP assignment (192.168.122.x range)
- No port forwarding needed - direct SSH access to VM IP addresses
- Standard virsh commands for VM management (start, shutdown, destroy, console)
- Integrates with existing libvirt infrastructure and monitoring tools
SSH Session Management:
- SSH keys are dynamically generated based on directory location using
format: ~/.ssh/kdevops-nixos-<directory>-<hash>
- This ensures unique keys per kdevops instance, preventing conflicts between multiple deployments
- SSH config entries are automatically managed during bringup
- Direct connections to DHCP-assigned IPs (no port forwarding complexity)
Path Compatibility:
- NixOS uses different system paths than traditional Linux distributions
- Python interpreter: /run/current-system/sw/bin/python3 (not /usr/bin/python3)
- Bash shell: /run/current-system/sw/bin/bash (not /bin/bash)
- Templates automatically detect and use correct paths for NixOS
VM Management:
- Professional VM lifecycle through libvirt system session
- XML-based VM configuration with proper resource allocation
- QCOW2 disk images with virtio drivers for performance
- Automatic network configuration via libvirt's default network
- Full integration with existing libvirt monitoring and management tools
Workflow Support:
The following workflows have initial NixOS support with only package dependency resolution, actual run time tests will be needed later:
- fstests: Filesystem testing (XFS, Btrfs, EXT4)
- blktests: Block layer testing (NVMe, SCSI, NBD)
- selftests: Linux kernel selftests
- mmtests: Memory management performance testing
- sysbench: Database performance benchmarking
- pynfs: NFS protocol testing
- ltp: Linux Test Project
- gitr: Git regression testing
Key Features:
- Declarative configuration through Nix expressions
- Reproducible builds using Nix flakes
- Automatic dependency resolution for workflows
- Directory-based isolation for multiple kdevops instances
- Full libvirt integration for professional VM management
- Automatic SSH configuration during bringup
- Standard networking without port forwarding complexity
This implementation provides a modern, functional approach to VM provisioning that leverages both NixOS's strengths in reproducibility and libvirt's professional VM management capabilities.
--- .gitignore | 3 + defconfigs/nixos | 27 + docs/kdevops-nixos.md | 404 ++++++++++++++ kconfigs/Kconfig.bringup | 22 +- kconfigs/Kconfig.nixos | 101 ++++ nixos/flake.nix | 32 ++ .../files/scripts/detect_libvirt_session.sh | 26 + playbooks/nixos.yml | 516 ++++++++++++++++++ .../devconfig/tasks/install-deps/main.yml | 1 + playbooks/roles/devconfig/tasks/main.yml | 4 +- playbooks/roles/gen_hosts/tasks/main.yml | 15 + .../roles/gen_hosts/templates/fstests.j2 | 20 + playbooks/roles/gen_hosts/templates/hosts.j2 | 16 + playbooks/roles/gen_nodes/tasks/main.yml | 24 + .../roles/gen_nodes/templates/nixos_nodes.j2 | 14 + .../roles/update_etc_hosts/tasks/main.yml | 2 + .../templates/nixos/configuration.nix.j2 | 138 +++++ playbooks/templates/nixos/flake.nix.j2 | 38 ++ .../nixos/hardware-configuration.nix.j2 | 42 ++ .../templates/nixos/run-vm-wrapper.sh.j2 | 159 ++++++ playbooks/templates/nixos/vm-libvirt.xml.j2 | 96 ++++ playbooks/templates/nixos/vms.nix.j2 | 45 ++ .../templates/nixos/workflow-deps.nix.j2 | 127 +++++ playbooks/update_ssh_config_nixos.yml | 57 ++ scripts/detect_libvirt_session.sh | 26 + scripts/nixos.Makefile | 93 ++++ scripts/nixos_ssh_key_name.py | 55 ++ scripts/provision.Makefile | 4 + scripts/status_nixos.sh | 57 ++ scripts/update_ssh_config_nixos.py | 133 +++++ 30 files changed, 2294 insertions(+), 3 deletions(-) create mode 100644 defconfigs/nixos create mode 100644 docs/kdevops-nixos.md create mode 100644 kconfigs/Kconfig.nixos create mode 100644 nixos/flake.nix create mode 100755 playbooks/files/scripts/detect_libvirt_session.sh create mode 100644 playbooks/nixos.yml create mode 100644 playbooks/roles/gen_nodes/templates/nixos_nodes.j2 create mode 100644 playbooks/templates/nixos/configuration.nix.j2 create mode 100644 playbooks/templates/nixos/flake.nix.j2 create mode 100644 playbooks/templates/nixos/hardware-configuration.nix.j2 create mode 100644 playbooks/templates/nixos/run-vm-wrapper.sh.j2 create mode 100644 playbooks/templates/nixos/vm-libvirt.xml.j2 create mode 100644 playbooks/templates/nixos/vms.nix.j2 create mode 100644 playbooks/templates/nixos/workflow-deps.nix.j2 create mode 100644 playbooks/update_ssh_config_nixos.yml create mode 100755 scripts/detect_libvirt_session.sh create mode 100644 scripts/nixos.Makefile create mode 100755 scripts/nixos_ssh_key_name.py create mode 100755 scripts/status_nixos.sh create mode 100755 scripts/update_ssh_config_nixos.py
.gitignore +3 -0
@@ -102,3 +102,6 @@ scripts/kconfig/.nconf-cfg Kconfig.passthrough_libvirt.generated archive/ + +# NixOS generated files +nixos/generated/
defconfigs/nixos new file +27 -0
@@ -0,0 +1,27 @@ +CONFIG_NIXOS=y +CONFIG_LIBVIRT=y + +# Disable mirror setup for NixOS +CONFIG_ENABLE_LOCAL_LINUX_MIRROR=n +CONFIG_USE_LOCAL_LINUX_MIRROR=n +CONFIG_INSTALL_LOCAL_LINUX_MIRROR=n +CONFIG_MIRROR_INSTALL=n + +CONFIG_NIXOS_USE_FLAKES=y +CONFIG_NIXOS_CHANNEL="nixos-unstable" +CONFIG_NIXOS_ENABLE_WORKFLOW_DEPS=y +CONFIG_NIXOS_LIBVIRT_SESSION_INFERENCE=y + +CONFIG_NIXOS_VM_MEMORY_MB=4096 +CONFIG_NIXOS_VM_DISK_SIZE_GB=20 +CONFIG_NIXOS_VM_VCPUS=4 + +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOW_LINUX_CUSTOM=y + +CONFIG_BOOTLINUX=y +CONFIG_BOOTLINUX_9P=n + +CONFIG_KDEVOPS_TRY_REFRESH_REPOS=y +CONFIG_KDEVOPS_TRY_UPDATE_SYSTEMS=y +CONFIG_KDEVOPS_TRY_INSTALL_KDEV_TOOLS=y
docs/kdevops-nixos.md new file +404 -0
404 lines not on this page · raw message
kconfigs/Kconfig.bringup +21 -1
@@ -5,6 +5,10 @@ config KDEVOPS_ENABLE_GUESTFS bool output yaml +config KDEVOPS_ENABLE_NIXOS + bool + output yaml + choice prompt "Node bring up method" default GUESTFS
@@ -39,6 +43,21 @@ config TERRAFORM If you are not using a cloud environment just disable this. +config NIXOS + bool "NixOS declarative configuration with libvirt" + select KDEVOPS_ENABLE_NIXOS + select EXTRA_STORAGE_SUPPORTS_512 + select EXTRA_STORAGE_SUPPORTS_1K + select EXTRA_STORAGE_SUPPORTS_2K + select EXTRA_STORAGE_SUPPORTS_4K + select EXTRA_STORAGE_SUPPORTS_LARGEIO + help + Use NixOS declarative configuration system to provision VMs with + libvirt. This provides a purely functional approach to VM management + with automatic dependency resolution based on enabled workflows. + NixOS will automatically infer the libvirt session type (system vs + user) based on your distribution's defaults, similar to guestfs. + config SKIP_BRINGUP bool "Skip bring up - bare metal or existing nodes" select EXTRA_STORAGE_SUPPORTS_512
@@ -55,10 +74,11 @@ endchoice config LIBVIRT bool - depends on GUESTFS + depends on GUESTFS || NIXOS default y source "kconfigs/Kconfig.guestfs" +source "kconfigs/Kconfig.nixos" source "terraform/Kconfig" if LIBVIRT source "kconfigs/Kconfig.libvirt"
kconfigs/Kconfig.nixos new file +101 -0
101 lines not on this page · raw message
nixos/flake.nix new file +32 -0
@@ -0,0 +1,32 @@ +{ + description = "kdevops NixOS VMs"; + + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + }; + + outputs = { self, nixpkgs }: { + nixosConfigurations = { + "kdevops" = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ + ./generated/configuration.nix + ./generated/hardware-configuration.nix + ./generated/workflow-deps.nix + ({ ... }: { + networking.hostName = "kdevops"; + }) + ]; + }; + }; + + # Build all VMs + defaultPackage.x86_64-linux = + nixpkgs.legacyPackages.x86_64-linux.writeShellScriptBin "build-vms" '' + echo "Building NixOS VMs..." + echo "Building kdevops..." + nix build .#nixosConfigurations.kdevops.config.system.build.vm + echo "All VMs built successfully!" + ''; + }; +}
playbooks/files/scripts/detect_libvirt_session.sh new file +26 -0
@@ -0,0 +1,26 @@ +#!/bin/bash +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Detect the appropriate libvirt session type (system vs user) based on +# distribution defaults, similar to how guestfs handles it. + +SCRIPTS_DIR=$(dirname $0) +source ${SCRIPTS_DIR}/libvirt_pool.sh + +OS_FILE="/etc/os-release" +LIBVIRT_URI="qemu:///system" # Default to system + +# Get the pool variables which includes distribution detection +get_pool_vars + +# Fedora defaults to user session +if [[ "$USES_QEMU_USER_SESSION" == "y" ]]; then + LIBVIRT_URI="qemu:///session" +fi + +# Override detection if explicitly configured +if [[ -n "$CONFIG_LIBVIRT_URI_PATH" ]]; then + LIBVIRT_URI="$CONFIG_LIBVIRT_URI_PATH" +fi + +echo "$LIBVIRT_URI"
playbooks/nixos.yml new file +516 -0
516 lines not on this page · raw message
playbooks/roles/devconfig/tasks/install-deps/main.yml +1 -0
@@ -22,6 +22,7 @@ - files: - "{{ ansible_facts['os_family'] | lower }}.yml" skip: true + when: ansible_facts['os_family'] != 'NixOS' tags: vars - name: Debian-specific setup
playbooks/roles/devconfig/tasks/main.yml +2 -2
@@ -197,7 +197,7 @@ chmod 755 {{ dev_bash_config }} fi args: - executable: /bin/bash + executable: "{{ '/run/current-system/sw/bin/bash' if (kdevops_enable_nixos | default(false)) else '/bin/bash' }}" when: dev_bash_config_file_copied is success - name: Copy the developer's favorite bash hacks over for root *if* it exists
@@ -224,7 +224,7 @@ chmod 755 {{ dev_bash_config_root }} fi args: - executable: /bin/bash + executable: "{{ '/run/current-system/sw/bin/bash' if (kdevops_enable_nixos | default(false)) else '/bin/bash' }}" when: dev_bash_config_file_copied_root is success - name: Check to see if system has GRUB2
playbooks/roles/gen_hosts/tasks/main.yml +15 -0
@@ -79,6 +79,20 @@ when: - not kdevops_workflows_dedicated_workflow - ansible_hosts_template.stat.exists + - not kdevops_enable_nixos|default(false)|bool + +- name: Generate the Ansible inventory file for NixOS + tags: ['hosts'] + ansible.builtin.template: + src: "{{ kdevops_hosts_template }}" + dest: "{{ ansible_cfg_inventory }}" + force: true + trim_blocks: True + lstrip_blocks: True + when: + - not kdevops_workflows_dedicated_workflow + - ansible_hosts_template.stat.exists + - kdevops_enable_nixos|default(false)|bool - name: Update Ansible inventory access modification time so make sees it updated ansible.builtin.file:
@@ -339,6 +353,7 @@ - kdevops_workflows_dedicated_workflow - kdevops_workflow_enable_fio_tests - ansible_hosts_template.stat.exists + - not kdevops_enable_nixos|default(false)|bool - name: Infer enabled mmtests test types
playbooks/roles/gen_hosts/templates/fstests.j2 +20 -0
@@ -1,10 +1,18 @@ [all] localhost ansible_connection=local {% for s in fstests_enabled_test_types %} +{% if kdevops_enable_nixos|default(false) %} +{{ kdevops_host_prefix }}-{{ s }} ansible_python_interpreter=/run/current-system/sw/bin/python3 +{% else %} {{ kdevops_host_prefix }}-{{ s }} +{% endif %} {% if kdevops_baseline_and_dev %} +{% if kdevops_enable_nixos|default(false) %} +{{ kdevops_host_prefix }}-{{ s }}-dev ansible_python_interpreter=/run/current-system/sw/bin/python3 +{% else %} {{ kdevops_host_prefix }}-{{ s }}-dev {% endif %} +{% endif %} {% endfor %} {% if kdevops_nfsd_enable %} {% if kdevops_loopback_nfs_enable %}
@@ -15,7 +23,11 @@ localhost ansible_connection=local ansible_python_interpreter = "{{ kdevops_python_interpreter }}" [baseline] {% for s in fstests_enabled_test_types %} +{% if kdevops_enable_nixos|default(false) %} +{{ kdevops_host_prefix }}-{{ s }} ansible_python_interpreter=/run/current-system/sw/bin/python3 +{% else %} {{ kdevops_host_prefix }}-{{ s }} +{% endif %} {% endfor %} {% if kdevops_nfsd_enable %} {% if kdevops_loopback_nfs_enable %}
@@ -27,7 +39,11 @@ ansible_python_interpreter = "{{ kdevops_python_interpreter }}" [dev] {% if kdevops_baseline_and_dev %} {% for s in fstests_enabled_test_types %} +{% if kdevops_enable_nixos|default(false) %} +{{ kdevops_host_prefix }}-{{ s }}-dev ansible_python_interpreter=/run/current-system/sw/bin/python3 +{% else %} {{ kdevops_host_prefix }}-{{ s }}-dev +{% endif %} {% endfor %} {% if kdevops_nfsd_enable %} {% if kdevops_loopback_nfs_enable %}
@@ -62,7 +78,11 @@ ansible_python_interpreter = "{{ kdevops_python_interpreter }}" ansible_python_interpreter = "{{ kdevops_python_interpreter }}" [krb5] {% for s in fstests_enabled_test_types %} +{% if kdevops_enable_nixos|default(false) %} +{{ kdevops_host_prefix }}-{{ s }} ansible_python_interpreter=/run/current-system/sw/bin/python3 +{% else %} {{ kdevops_host_prefix }}-{{ s }} +{% endif %} {% endfor %} {% if kdevops_nfsd_enable %} {{ kdevops_hosts_prefix }}-nfsd
playbooks/roles/gen_hosts/templates/hosts.j2 +16 -0
@@ -184,10 +184,18 @@ write-your-own-template-for-your-workflow-and-task {% else %} [all] localhost ansible_connection=local +{% if kdevops_enable_nixos|default(false) %} +{{ kdevops_host_prefix }} ansible_python_interpreter=/run/current-system/sw/bin/python3 +{% else %} {{ kdevops_host_prefix }} +{% endif %} {% if kdevops_baseline_and_dev == True %} +{% if kdevops_enable_nixos|default(false) %} +{{ kdevops_host_prefix }}-dev ansible_python_interpreter=/run/current-system/sw/bin/python3 +{% else %} {{ kdevops_host_prefix }}-dev {% endif %} +{% endif %} {% if kdevops_enable_iscsi %} {{ kdevops_host_prefix }}-iscsi {% endif %}
@@ -197,13 +205,21 @@ localhost ansible_connection=local [all:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" [baseline] +{% if kdevops_enable_nixos|default(false) %} +{{ kdevops_host_prefix }} ansible_python_interpreter=/run/current-system/sw/bin/python3 +{% else %} {{ kdevops_host_prefix }} +{% endif %} [baseline:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" [dev] {% if kdevops_baseline_and_dev %} +{% if kdevops_enable_nixos|default(false) %} +{{ kdevops_host_prefix }}-dev ansible_python_interpreter=/run/current-system/sw/bin/python3 +{% else %} {{ kdevops_host_prefix }}-dev {% endif %} +{% endif %} [dev:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" {% if kdevops_enable_iscsi %}
playbooks/roles/gen_nodes/tasks/main.yml +24 -0
@@ -27,6 +27,12 @@ mode: "0755" when: kdevops_enable_guestfs|bool +- name: Create nixos directory + ansible.builtin.file: + path: "{{ nixos_config_dir }}" + state: directory + when: kdevops_enable_nixos | default(false) | bool + - name: Verify Ansible nodes template file exists {{ kdevops_nodes_template_full_path }} ansible.builtin.stat: path: "{{ kdevops_nodes_template_full_path }}"
@@ -148,6 +154,23 @@ mode: "0644" when: - not kdevops_workflows_dedicated_workflow + - ansible_nodes_template.stat.exists + - not kdevops_enable_nixos|default(false)|bool + +- name: Generate the NixOS kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template + tags: ['nodes'] + vars: + node_template: "{{ kdevops_nodes_template | basename }}" + all_generic_nodes: "{{ generic_nodes }}" + nodes: "{{ all_generic_nodes }}" + ansible.builtin.template: + src: "{{ node_template }}" + dest: "{{ topdir_path }}/{{ kdevops_nodes }}" + force: true + when: + - not kdevops_workflows_dedicated_workflow + - ansible_nodes_template.stat.exists + - kdevops_enable_nixos|default(false)|bool - name: Generate the builder kdevops nodes file using nodes file using template as jinja2 source template
@@ -162,6 +185,7 @@ force: true when: - bootlinux_builder + - ansible_nodes_template.stat.exists - name: Generate the pynfs kdevops nodes file using nodes file using template as jinja2 source template
playbooks/roles/gen_nodes/templates/nixos_nodes.j2 new file +14 -0
@@ -0,0 +1,14 @@ +--- +# Ansible nodes file generated for NixOS VMs + +{% for node in nodes %} +{{ node }}: + ansible_host: 192.168.100.{{ loop.index + 1 }} + ansible_user: kdevops + ansible_ssh_private_key_file: {{ topdir_path }}/.ssh/kdevops_id_rsa + ansible_python_interpreter: /run/current-system/sw/bin/python3 + vm_name: {{ node }} + vm_memory_mb: {{ nixos_vm_memory_mb }} + vm_vcpus: {{ nixos_vm_vcpus }} + vm_disk_size_gb: {{ nixos_vm_disk_size_gb }} +{% endfor %}
playbooks/roles/update_etc_hosts/tasks/main.yml +2 -0
@@ -57,6 +57,7 @@ with_items: "{{ ueh_hosts }}" when: - terraform_private_net_enabled + - not (kdevops_enable_nixos | default(false)) - name: Add IP address of all hosts to all hosts become: true
@@ -69,6 +70,7 @@ with_items: "{{ ueh_hosts }}" when: - not terraform_private_net_enabled + - not (kdevops_enable_nixos | default(false)) - name: Fix up hostname on Debian guestfs hosts become: true
playbooks/templates/nixos/configuration.nix.j2 new file +138 -0
138 lines not on this page · raw message
playbooks/templates/nixos/flake.nix.j2 new file +38 -0
@@ -0,0 +1,38 @@ +{ + description = "kdevops NixOS VMs"; + + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/{{ nixos_channel }}"; + }; + + outputs = { self, nixpkgs }: { + nixosConfigurations = { +{% for node in groups['all'] if node != 'localhost' %} + "{{ node }}" = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ + ./generated/configuration.nix + ./generated/hardware-configuration.nix +{% if nixos_enable_workflow_deps %} + ./generated/workflow-deps.nix +{% endif %} + ({ ... }: { + networking.hostName = "{{ node }}"; + }) + ]; + }; +{% endfor %} + }; + + # Build all VMs + defaultPackage.x86_64-linux = + nixpkgs.legacyPackages.x86_64-linux.writeShellScriptBin "build-vms" '' + echo "Building NixOS VMs..." +{% for node in groups['all'] if node != 'localhost' %} + echo "Building {{ node }}..." + nix build .#nixosConfigurations.{{ node }}.config.system.build.vm +{% endfor %} + echo "All VMs built successfully!" + ''; + }; +}
playbooks/templates/nixos/hardware-configuration.nix.j2 new file +42 -0
42 lines not on this page · raw message
playbooks/templates/nixos/run-vm-wrapper.sh.j2 new file +159 -0
159 lines not on this page · raw message
playbooks/templates/nixos/vm-libvirt.xml.j2 new file +96 -0
96 lines not on this page · raw message
playbooks/templates/nixos/vms.nix.j2 new file +45 -0
45 lines not on this page · raw message
playbooks/templates/nixos/workflow-deps.nix.j2 new file +127 -0
127 lines not on this page · raw message
playbooks/update_ssh_config_nixos.yml new file +57 -0
57 lines not on this page · raw message
scripts/detect_libvirt_session.sh new file +26 -0
@@ -0,0 +1,26 @@ +#!/bin/bash +# SPDX-License-Identifier: copyleft-next-0.3.1 +# +# Detect the appropriate libvirt session type (system vs user) based on +# distribution defaults, similar to how guestfs handles it. + +SCRIPTS_DIR=$(dirname $0) +source ${SCRIPTS_DIR}/libvirt_pool.sh + +OS_FILE="/etc/os-release" +LIBVIRT_URI="qemu:///system" # Default to system + +# Get the pool variables which includes distribution detection +get_pool_vars + +# Fedora defaults to user session +if [[ "$USES_QEMU_USER_SESSION" == "y" ]]; then + LIBVIRT_URI="qemu:///session" +fi + +# Override detection if explicitly configured +if [[ -n "$CONFIG_LIBVIRT_URI_PATH" ]]; then + LIBVIRT_URI="$CONFIG_LIBVIRT_URI_PATH" +fi + +echo "$LIBVIRT_URI"
scripts/nixos.Makefile new file +93 -0
93 lines not on this page · raw message
scripts/nixos_ssh_key_name.py new file +55 -0
55 lines not on this page · raw message
scripts/provision.Makefile +4 -0
@@ -60,6 +60,10 @@ ifeq (y,$(CONFIG_GUESTFS)) include scripts/guestfs.Makefile endif +ifeq (y,$(CONFIG_NIXOS)) +include scripts/nixos.Makefile +endif + KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_SSH) KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_DEVCONFIG)
scripts/status_nixos.sh new file +57 -0
57 lines not on this page · raw message
scripts/update_ssh_config_nixos.py new file +133 -0
133 lines not on this page · raw message
Signature · 1 line
2.50.1
PATCH 3/3 mirror: add Nix binary cache mirroring support
In reply to Luis ChamberlainAdd support for a Nix binary cache mirroring infrastructure for enterprise environments to reduce latencies and public bandwidth consumption for CIs. This dramatically reduces NixOS package download times from hundreds of packages to local cache hits.
Key features:
- nginx-based proxy caching for Nix binary cache
- Automatic mirror detection for client-side configuration
- systemd service and timer for cache synchronization
- Integration with existing defconfig-mirror infrastructure
- Client auto-configuration when local mirrors are available
Components:
- playbooks/roles/nix-cache-mirror/: Complete Ansible role
- scripts/check_nix_mirror.sh: Mirror detection and URL discovery
- kconfigs/Kconfig.mirror: Mirror configuration options
- Integration with defconfig-mirror for one-time setup
The mirror setup follows existing kdevops mirror patterns and integrates with the git mirror infrastructure already in place.
--- defconfigs/mirror | 1 + kconfigs/Kconfig.mirror | 68 +++++++- kconfigs/Kconfig.nixos | 29 ++++ playbooks/linux-mirror.yml | 2 + playbooks/nixos.yml | 28 ++- .../linux-mirror/python/start-mirroring.py | 29 +++- .../roles/nix-cache-mirror/defaults/main.yml | 7 + .../roles/nix-cache-mirror/handlers/main.yml | 11 ++ .../roles/nix-cache-mirror/tasks/main.yml | 161 ++++++++++++++++++ .../templates/nix-cache-mirror-cache.conf.j2 | 3 + .../templates/nix-cache-mirror.nginx.j2 | 81 +++++++++ .../templates/nix-cache-sync.service.j2 | 17 ++ .../templates/nix-cache-sync.timer.j2 | 11 ++ scripts/check_nix_mirror.sh | 66 +++++++ 14 files changed, 507 insertions(+), 7 deletions(-) create mode 100644 playbooks/roles/nix-cache-mirror/defaults/main.yml create mode 100644 playbooks/roles/nix-cache-mirror/handlers/main.yml create mode 100644 playbooks/roles/nix-cache-mirror/tasks/main.yml create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2 create mode 100644 playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2 create mode 100755 scripts/check_nix_mirror.sh
defconfigs/mirror +1 -0
@@ -2,3 +2,4 @@ CONFIG_SKIP_BRINGUP=y CONFIG_WORKFLOWS=n CONFIG_INSTALL_LOCAL_LINUX_MIRROR=y CONFIG_LINUX_MIRROR_NFS=y +CONFIG_INSTALL_NIX_CACHE_MIRROR=y
kconfigs/Kconfig.mirror +66 -2
85 lines not on this page · raw message
kconfigs/Kconfig.nixos +29 -0
@@ -91,6 +91,35 @@ config NIXOS_SSH_PORT help SSH port to use for connecting to NixOS VMs. +config NIXOS_USE_LOCAL_MIRROR + bool "Use local Nix binary cache mirror" + output yaml + default $(shell, scripts/check_nix_mirror.sh USE_NIX_CACHE_MIRROR) + help + Enable this to use a local Nix binary cache mirror if available. + This dramatically speeds up NixOS builds by avoiding repeated + downloads of packages from cache.nixos.org. + + When enabled, kdevops will automatically detect and configure + VMs to use the local mirror if available at /mirror/nix-cache + or via HTTP on port 8080. + + This requires that you have set up a mirror server using the + defconfig-mirror configuration. + +config NIXOS_MIRROR_URL + string "Nix binary cache mirror URL" + output yaml + default $(shell, scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL) + depends on NIXOS_USE_LOCAL_MIRROR + help + The URL of the local Nix binary cache mirror. This is + automatically detected based on available mirrors: + - HTTP mirror if running on port 8080 + - File-based mirror at /mirror/nix-cache (for NFS mounts) + + Leave empty to auto-detect at runtime. + config NIXOS_DEBUG_MODE bool "Enable debug mode for NixOS provisioning" default n
playbooks/linux-mirror.yml +2 -0
@@ -3,3 +3,5 @@ hosts: localhost roles: - role: linux-mirror + - role: nix-cache-mirror + when: install_nix_cache_mirror | default(false) | bool
playbooks/nixos.yml +26 -2
@@ -95,6 +95,21 @@ ansible.builtin.set_fact: nixos_ssh_authorized_key: "{{ ssh_public_key['content'] | b64decode | trim }}" + - name: Detect local Nix cache mirror URL if enabled + ansible.builtin.shell: | + bash {{ playbook_dir }}/../scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL + register: detected_mirror_url + when: nixos_use_local_mirror | default(false) | bool and (nixos_mirror_url is not defined or nixos_mirror_url == "") + changed_when: false + + - name: Set detected mirror URL + ansible.builtin.set_fact: + nixos_mirror_url: "{{ detected_mirror_url.stdout | trim }}" + when: + - detected_mirror_url is defined + - detected_mirror_url.stdout is defined + - detected_mirror_url.stdout | trim != "" + - name: Template base NixOS configuration ansible.builtin.template: src: nixos/configuration.nix.j2
@@ -176,9 +191,18 @@ fi # Configure Nix to use local mirror if available - {% if nixos_use_local_mirror is defined and nixos_use_local_mirror and nixos_mirror_url is defined and nixos_mirror_url != "" %} + {% if nixos_use_local_mirror is defined and nixos_use_local_mirror %} + # Dynamically detect mirror URL if not provided + {% if nixos_mirror_url is not defined or nixos_mirror_url == "" %} + DETECTED_MIRROR=$(bash {{ playbook_dir }}/../scripts/check_nix_mirror.sh NIX_CACHE_MIRROR_URL) + if [ -n "$DETECTED_MIRROR" ]; then + export NIX_CONFIG="substituters = $DETECTED_MIRROR https://cache.nixos.org" + echo "Using detected local Nix cache mirror: $DETECTED_MIRROR" + fi + {% else %} export NIX_CONFIG="substituters = {{ nixos_mirror_url }} https://cache.nixos.org" - echo "Using local Nix cache mirror: {{ nixos_mirror_url }}" + echo "Using configured local Nix cache mirror: {{ nixos_mirror_url }}" + {% endif %} {% endif %} cd {{ nixos_generation_dir }}
playbooks/roles/linux-mirror/python/start-mirroring.py +26 -3
@@ -54,7 +54,8 @@ def mirror_entry(mirror, args): cmd = cmd + reference_args mirror_target = mirror_path + target if os.path.isdir(mirror_target): - return + sys.stdout.write("Skipping %s - mirror already exists at %s\n" % (short_name, mirror_target)) + return "skipped" sys.stdout.write("Mirroring: %s onto %s\n" % (short_name, mirror_target)) if args.verbose: sys.stdout.write("%s\n" % (cmd))
@@ -74,6 +75,7 @@ def mirror_entry(mirror, args): process.wait() if process.returncode != 0: raise Exception(f"Failed clone with:\n%s" % (" ".join(cmd))) + return "cloned" def main():
@@ -134,15 +136,36 @@ def main(): % (mirror.get("short_name"), args.yaml_mirror, total) ) + # Statistics tracking + stats = {"skipped": 0, "cloned": 0, "failed": 0} # Mirror trees without a reference first for mirror in yaml_vars["mirrors"]: if not mirror.get("reference"): - mirror_entry(mirror, args) + result = mirror_entry(mirror, args) + if result == "skipped": + stats["skipped"] += 1 + elif result == "cloned": + stats["cloned"] += 1 + else: + stats["failed"] += 1 # Mirror trees which need a reference last for mirror in yaml_vars["mirrors"]: if mirror.get("reference"): - mirror_entry(mirror, args) + result = mirror_entry(mirror, args) + if result == "skipped": + stats["skipped"] += 1 + elif result == "cloned": + stats["cloned"] += 1 + else: + stats["failed"] += 1 + # Print summary + sys.stdout.write("\n=== Mirror Summary ===\n") + sys.stdout.write("Total repositories: %d\n" % total) + sys.stdout.write("Skipped (already exist): %d\n" % stats["skipped"]) + sys.stdout.write("Cloned (new): %d\n" % stats["cloned"]) + if stats["failed"] > 0: + sys.stdout.write("Failed: %d\n" % stats["failed"]) if __name__ == "__main__":
playbooks/roles/nix-cache-mirror/defaults/main.yml new file +7 -0
@@ -0,0 +1,7 @@ +--- +# Default variables for nix-cache-mirror role +nix_cache_mirror_path: "/mirror/nix-cache" +nix_cache_mirror_port: 8080 +nix_cache_upstream_url: "https://cache.nixos.org" +nix_cache_mirror_nginx_conf_path: "/etc/nginx/sites-available/nix-cache-mirror" +nix_cache_mirror_nginx_enabled_path: "/etc/nginx/sites-enabled/nix-cache-mirror"
playbooks/roles/nix-cache-mirror/handlers/main.yml new file +11 -0
@@ -0,0 +1,11 @@ +--- +- name: reload nginx + become: true + ansible.builtin.systemd: + name: nginx + state: reloaded + +- name: reload systemd + become: true + ansible.builtin.systemd: + daemon_reload: true
playbooks/roles/nix-cache-mirror/tasks/main.yml new file +161 -0
161 lines not on this page · raw message
playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror-cache.conf.j2 new file +3 -0
@@ -0,0 +1,3 @@ +# Nix cache mirror proxy cache configuration +# This file must be included at http context level +proxy_cache_path {{ nix_cache_mirror_path }}/nginx-cache levels=1:2 keys_zone=nixcache:100m max_size=50g inactive=7d use_temp_path=off;
playbooks/roles/nix-cache-mirror/templates/nix-cache-mirror.nginx.j2 new file +81 -0
81 lines not on this page · raw message
playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.service.j2 new file +17 -0
@@ -0,0 +1,17 @@ +[Unit] +Description=Nix Binary Cache Sync Service +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=www-data +Group=www-data +ExecStart=/usr/bin/curl -f -s "{{ mirror_nix_cache_upstream_url | default('https://cache.nixos.org') }}/nix-cache-info" -o "{{ nix_cache_mirror_path }}/nix-cache-info" +ExecStartPost=/bin/bash -c 'find {{ nix_cache_mirror_path }} -name "*.narinfo" -mtime +7 -delete' +ExecStartPost=/bin/bash -c 'find {{ nix_cache_mirror_path }} -name "*.nar*" -mtime +7 -delete' +StandardOutput=journal +StandardError=journal + +[Install] +WantedBy=multi-user.target
playbooks/roles/nix-cache-mirror/templates/nix-cache-sync.timer.j2 new file +11 -0
@@ -0,0 +1,11 @@ +[Unit] +Description=Run Nix Binary Cache Sync Service hourly +Requires=nix-cache-sync.service + +[Timer] +OnCalendar=hourly +Persistent=true +RandomizedDelaySec=300 + +[Install] +WantedBy=timers.target
scripts/check_nix_mirror.sh new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
2.50.1
Quoted 7 lines · Luis Chamberlain
This adds a barebones nixos support. Enough to get ssh going and initial mirror support. The mirror works but its empty as it has no packages. So all we can do is bringup. Enough to kick the tires off. Some initial packages are added in preparation for future workflows suppor but that is a bit more complex and requires moer work. This should be enough to let the ball get rolling.
Quoted 5 lines
Luis Chamberlain (3):
common: use fallback for group inference on remote systems
nixos: add NixOS support as third bringup option with libvirt
integration
mirror: add Nix binary cache mirroring support
Pushed.
Luis