Skip to contentSkip to conversation

terraform/OCI: Grab secrets from ~/.oci/config

RFC PATCH In review [email protected] 3 participants 11 messages Apr 3, 2025 – Apr 4, 2025

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 terraform/OCI: Grab secrets from ~/.oci/config

From: Chuck Lever <[email protected]>

Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.

terraform's API authentication is now managed outside of Kconfig, as is done with AWS. An update to docs/kdevops-terraform.md to follow.

---
 .../templates/oci/terraform.tfvars.j2         |  5 +---
 scripts/terraform.Makefile                    |  4 ---
 terraform/oci/kconfigs/Kconfig.identity       | 27 +++++++------------
 terraform/oci/provider.tf                     |  7 ++---
 terraform/oci/vars.tf                         | 25 ++++-------------
 5 files changed, 17 insertions(+), 51 deletions(-)

The tenancy OCID, user OCID, fingerprint, and private key path Kconfig settings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.

playbooks/roles/gen_tfvars/templates/oci/terraform.tfvars.j2 +1 -4
index f7b9ffb48c64..73b963230882 100644
@@ -1,8 +1,5 @@
 oci_region = "{{ terraform_oci_region }}"
-oci_tenancy_ocid = "{{ terraform_oci_tenancy_ocid }}"
-oci_user_ocid = "{{ terraform_oci_user_ocid }}"
-oci_user_private_key_path = "{{ terraform_oci_user_private_key_path }}"
-oci_user_fingerprint = "{{ terraform_oci_user_fingerprint }}"
+oci_config_file_profile = "{{ terraform_oci_config_file_profile }}"
 oci_availablity_domain = "{{ terraform_oci_availablity_domain }}"
 oci_compartment_ocid = "{{ terraform_oci_compartment_ocid }}"
 oci_shape = "{{ terraform_oci_shape }}"
scripts/terraform.Makefile +0 -4
index 3235c1af84b9..75ba9b97b518 100644
@@ -96,10 +96,6 @@ endif
 
 ifeq (y,$(CONFIG_TERRAFORM_OCI))
 TERRAFORM_EXTRA_VARS += terraform_oci_region=$(subst ",,$(CONFIG_TERRAFORM_OCI_REGION))
-TERRAFORM_EXTRA_VARS += terraform_oci_tenancy_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_TENANCY_OCID))
-TERRAFORM_EXTRA_VARS += terraform_oci_user_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_OCID))
-TERRAFORM_EXTRA_VARS += terraform_oci_user_private_key_path=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_PRIVATE_KEY_PATH))
-TERRAFORM_EXTRA_VARS += terraform_oci_user_fingerprint=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_FINGERPRINT))
 TERRAFORM_EXTRA_VARS += terraform_oci_availablity_domain=$(subst ",,$(CONFIG_TERRAFORM_OCI_AVAILABLITY_DOMAIN))
 TERRAFORM_EXTRA_VARS += terraform_oci_compartment_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_COMPARTMENT_OCID))
 TERRAFORM_EXTRA_VARS += terraform_oci_shape=$(subst ",,$(CONFIG_TERRAFORM_OCI_SHAPE))
terraform/oci/kconfigs/Kconfig.identity +9 -18
index 928c700d9bd4..dce8f3bb54d9 100644
@@ -5,23 +5,14 @@ config TERRAFORM_OCI_TENANCY_OCID
 	  Read this:
 	  https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm
 
-config TERRAFORM_OCI_USER_OCID
-	string "OCI user OCID"
+config TERRAFORM_OCI_CONFIG_FILE_PROFILE
+	string "Authentication profile entry"
+	output yaml
+	default "DEFAULT"
 	help
-	  The OCI ID of the user to use.
-	  Read this:
-	  https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm
+	  Authentication secrets are stored in ~/.oci/config. That
+	  file may contain more than one entry/identity. Select the
+	  entry in ~/.oci/config that kdevops uses to
+	  authenticate to OCI. Examples:
 
-config TERRAFORM_OCI_USER_PRIVATE_KEY_PATH
-	string "OCI user private key path"
-	help
-	  Path to the file containing the private key of the user.
-	  Read this:
-	  https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm
-
-config TERRAFORM_OCI_USER_FINGERPRINT
-	string "OCI user fingerprint"
-	help
-	  Fingerprint for the key pair being used.
-	  Read this:
-	  https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm
+	  https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm
terraform/oci/provider.tf +2 -5
index 80361d3ac562..c4619e244d73 100644
@@ -2,9 +2,6 @@ terraform {
 }
 
 provider "oci" {
-  tenancy_ocid			= var.oci_tenancy_ocid
-  user_ocid			= var.oci_user_ocid
-  private_key_path		= var.oci_user_private_key_path
-  fingerprint			= var.oci_user_fingerprint
-  region			= var.oci_region
+  config_file_profile = var.oci_config_file_profile
+  region              = var.oci_region
 }
terraform/oci/vars.tf +5 -20
index 4c6383d9b231..9e67ee9a8959 100644
@@ -1,28 +1,13 @@
+variable "oci_config_file_profile" {
+  description = "Entry in ~/.oci/config to use for API authentication"
+  type        = string
+}
+
 variable "oci_region" {
   description = "An OCI region"
   type        = string
 }
 
-variable "oci_tenancy_ocid" {
-  description = "OCID of your tenancy"
-  type        = string
-}
-
-variable "oci_user_ocid" {
-  description = "OCID of the user calling the API"
-  type        = string
-}
-
-variable "oci_user_private_key_path" {
-  description = "The path of the private key stored on your computer"
-  type        = string
-}
-
-variable "oci_user_fingerprint" {
-  description = "Fingerprint for the key pair being used"
-  type        = string
-}
-
 variable "oci_availablity_domain" {
   description = "Name of availability domain"
   type        = string
Signature · 1 line
2.48.1
#1 lore raw
Quoted 20 lines · On 4/3/25 10:49 AM, [email protected] wrote:

From: Chuck Lever <[email protected]>

Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.

terraform's API authentication is now managed outside of Kconfig, as is done with AWS. An update to docs/kdevops-terraform.md to follow.

---
 .../templates/oci/terraform.tfvars.j2         |  5 +---
 scripts/terraform.Makefile                    |  4 ---
 terraform/oci/kconfigs/Kconfig.identity       | 27 +++++++------------
 terraform/oci/provider.tf                     |  7 ++---
 terraform/oci/vars.tf                         | 25 ++++-------------
 5 files changed, 17 insertions(+), 51 deletions(-)
Quoted 4 lines

The tenancy OCID, user OCID, fingerprint, and private key path Kconfig settings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.

It appears that, though undocumented, terraform's azurerm provider can also pull its authentication material from a home directory dot file (~/.azure/azureProfile.json). Proof-of-concept tested and working.

Google already works this way. So we can use the "secrets are stored outside of kdevops' .config file" for all four major cloud providers.

Quoted 120 lines
playbooks/roles/gen_tfvars/templates/oci/terraform.tfvars.j2 +1 -4
index f7b9ffb48c64..73b963230882 100644
@@ -1,8 +1,5 @@
 oci_region = "{{ terraform_oci_region }}"
-oci_tenancy_ocid = "{{ terraform_oci_tenancy_ocid }}"
-oci_user_ocid = "{{ terraform_oci_user_ocid }}"
-oci_user_private_key_path = "{{ terraform_oci_user_private_key_path }}"
-oci_user_fingerprint = "{{ terraform_oci_user_fingerprint }}"
+oci_config_file_profile = "{{ terraform_oci_config_file_profile }}"
 oci_availablity_domain = "{{ terraform_oci_availablity_domain }}"
 oci_compartment_ocid = "{{ terraform_oci_compartment_ocid }}"
 oci_shape = "{{ terraform_oci_shape }}"
scripts/terraform.Makefile +0 -4
index 3235c1af84b9..75ba9b97b518 100644
@@ -96,10 +96,6 @@ endif
 
 ifeq (y,$(CONFIG_TERRAFORM_OCI))
 TERRAFORM_EXTRA_VARS += terraform_oci_region=$(subst ",,$(CONFIG_TERRAFORM_OCI_REGION))
-TERRAFORM_EXTRA_VARS += terraform_oci_tenancy_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_TENANCY_OCID))
-TERRAFORM_EXTRA_VARS += terraform_oci_user_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_OCID))
-TERRAFORM_EXTRA_VARS += terraform_oci_user_private_key_path=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_PRIVATE_KEY_PATH))
-TERRAFORM_EXTRA_VARS += terraform_oci_user_fingerprint=$(subst ",,$(CONFIG_TERRAFORM_OCI_USER_FINGERPRINT))
 TERRAFORM_EXTRA_VARS += terraform_oci_availablity_domain=$(subst ",,$(CONFIG_TERRAFORM_OCI_AVAILABLITY_DOMAIN))
 TERRAFORM_EXTRA_VARS += terraform_oci_compartment_ocid=$(subst ",,$(CONFIG_TERRAFORM_OCI_COMPARTMENT_OCID))
 TERRAFORM_EXTRA_VARS += terraform_oci_shape=$(subst ",,$(CONFIG_TERRAFORM_OCI_SHAPE))
terraform/oci/kconfigs/Kconfig.identity +9 -18
index 928c700d9bd4..dce8f3bb54d9 100644
@@ -5,23 +5,14 @@ config TERRAFORM_OCI_TENANCY_OCID
 	  Read this:
 	  https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm
 
-config TERRAFORM_OCI_USER_OCID
-	string "OCI user OCID"
+config TERRAFORM_OCI_CONFIG_FILE_PROFILE
+	string "Authentication profile entry"
+	output yaml
+	default "DEFAULT"
 	help
-	  The OCI ID of the user to use.
-	  Read this:
-	  https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm
+	  Authentication secrets are stored in ~/.oci/config. That
+	  file may contain more than one entry/identity. Select the
+	  entry in ~/.oci/config that kdevops uses to
+	  authenticate to OCI. Examples:
 
-config TERRAFORM_OCI_USER_PRIVATE_KEY_PATH
-	string "OCI user private key path"
-	help
-	  Path to the file containing the private key of the user.
-	  Read this:
-	  https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm
-
-config TERRAFORM_OCI_USER_FINGERPRINT
-	string "OCI user fingerprint"
-	help
-	  Fingerprint for the key pair being used.
-	  Read this:
-	  https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/terraformproviderconfiguration.htm
+	  https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm
terraform/oci/provider.tf +2 -5
index 80361d3ac562..c4619e244d73 100644
@@ -2,9 +2,6 @@ terraform {
 }
 
 provider "oci" {
-  tenancy_ocid			= var.oci_tenancy_ocid
-  user_ocid			= var.oci_user_ocid
-  private_key_path		= var.oci_user_private_key_path
-  fingerprint			= var.oci_user_fingerprint
-  region			= var.oci_region
+  config_file_profile = var.oci_config_file_profile
+  region              = var.oci_region
 }
terraform/oci/vars.tf +5 -20
index 4c6383d9b231..9e67ee9a8959 100644
@@ -1,28 +1,13 @@
+variable "oci_config_file_profile" {
+  description = "Entry in ~/.oci/config to use for API authentication"
+  type        = string
+}
+
 variable "oci_region" {
   description = "An OCI region"
   type        = string
 }
 
-variable "oci_tenancy_ocid" {
-  description = "OCID of your tenancy"
-  type        = string
-}
-
-variable "oci_user_ocid" {
-  description = "OCID of the user calling the API"
-  type        = string
-}
-
-variable "oci_user_private_key_path" {
-  description = "The path of the private key stored on your computer"
-  type        = string
-}
-
-variable "oci_user_fingerprint" {
-  description = "Fingerprint for the key pair being used"
-  type        = string
-}
-
 variable "oci_availablity_domain" {
   description = "Name of availability domain"
   type        = string
Signature · 1 line
Chuck Lever
#2 lore raw
Quoted 21 lines · Chuck Lever
Quoted 20 lines · On 4/3/25 10:49 AM, [email protected] wrote:

From: Chuck Lever <[email protected]>

Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.

terraform's API authentication is now managed outside of Kconfig, as is done with AWS. An update to docs/kdevops-terraform.md to follow.

---
 .../templates/oci/terraform.tfvars.j2         |  5 +---
 scripts/terraform.Makefile                    |  4 ---
 terraform/oci/kconfigs/Kconfig.identity       | 27 +++++++------------
 terraform/oci/provider.tf                     |  7 ++---
 terraform/oci/vars.tf                         | 25 ++++-------------
 5 files changed, 17 insertions(+), 51 deletions(-)
Quoted 8 lines
Quoted 4 lines

The tenancy OCID, user OCID, fingerprint, and private key path Kconfig settings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.

It appears that, though undocumented, terraform's azurerm provider can also pull its authentication material from a home directory dot file (~/.azure/azureProfile.json). Proof-of-concept tested and working.

Nice!

Quoted 2 lines

Google already works this way. So we can use the "secrets are stored outside of kdevops' .config file" for all four major cloud providers.

Do we want a Kconfig SUPPORTS_SEPARATE_CLOUD_SECRETS or do we want to just implicate this is a requirement?

Luis

#3 lore raw
Quoted 31 lines · Luis Chamberlain
Quoted 21 lines · Chuck Lever
Quoted 20 lines · On 4/3/25 10:49 AM, [email protected] wrote:

From: Chuck Lever <[email protected]>

Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.

terraform's API authentication is now managed outside of Kconfig, as is done with AWS. An update to docs/kdevops-terraform.md to follow.

---
 .../templates/oci/terraform.tfvars.j2         |  5 +---
 scripts/terraform.Makefile                    |  4 ---
 terraform/oci/kconfigs/Kconfig.identity       | 27 +++++++------------
 terraform/oci/provider.tf                     |  7 ++---
 terraform/oci/vars.tf                         | 25 ++++-------------
 5 files changed, 17 insertions(+), 51 deletions(-)
Quoted 8 lines
Quoted 4 lines

The tenancy OCID, user OCID, fingerprint, and private key path Kconfig settings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.

It appears that, though undocumented, terraform's azurerm provider can also pull its authentication material from a home directory dot file (~/.azure/azureProfile.json). Proof-of-concept tested and working.

Quoted 7 lines

Nice!

Quoted 2 lines

Google already works this way. So we can use the "secrets are stored outside of kdevops' .config file" for all four major cloud providers.

Do we want a Kconfig SUPPORTS_SEPARATE_CLOUD_SECRETS or do we want to just implicate this is a requirement?

IMHO "separate secrets" is just more secure overall, and I generally lean towards not enabling users to select a lower level of security unless we absolutely have to.

- kdevops in Google and AWS already support only separate secrets
- kdevops in Azure was not working (for other reasons) so there's no backwards compatibility issue if we change it

Only possible issue is with OCI, and this series makes enough changes that anyone who upgrades kdevops will need to rework their .config anyway.

The series has a patch at the end (maybe I didn't post that) that updates docs/kdevops-terraform.md.

On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.

Signature · 1 line
Chuck Lever
#4 lore raw
Quoted 37 lines · Chuck Lever
Quoted 36 lines · Luis Chamberlain
Quoted 21 lines · Chuck Lever
Quoted 20 lines · On 4/3/25 10:49 AM, [email protected] wrote:

From: Chuck Lever <[email protected]>

Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.

terraform's API authentication is now managed outside of Kconfig, as is done with AWS. An update to docs/kdevops-terraform.md to follow.

---
 .../templates/oci/terraform.tfvars.j2         |  5 +---
 scripts/terraform.Makefile                    |  4 ---
 terraform/oci/kconfigs/Kconfig.identity       | 27 +++++++------------
 terraform/oci/provider.tf                     |  7 ++---
 terraform/oci/vars.tf                         | 25 ++++-------------
 5 files changed, 17 insertions(+), 51 deletions(-)
Quoted 8 lines
Quoted 4 lines

The tenancy OCID, user OCID, fingerprint, and private key path Kconfig settings would no longer be needed. This patch fits somewhere in the middle of the 00/31 series, replacing several of those patches.

It appears that, though undocumented, terraform's azurerm provider can also pull its authentication material from a home directory dot file (~/.azure/azureProfile.json). Proof-of-concept tested and working.

Nice!

Quoted 2 lines

Google already works this way. So we can use the "secrets are stored outside of kdevops' .config file" for all four major cloud providers.

Quoted 6 lines
Quoted 2 lines

Do we want a Kconfig SUPPORTS_SEPARATE_CLOUD_SECRETS or do we want to just implicate this is a requirement?

IMHO "separate secrets" is just more secure overall, and I generally lean towards not enabling users to select a lower level of security unless we absolutely have to.

I'm all in favor of simply making a rule to never allow secrets on .config. That lets us scale for CIs and enable reprodducible ci-workloads safely and collection of results on kdevops-results-archive.

Quoted 3 lines

- kdevops in Google and AWS already support only separate secrets
- kdevops in Azure was not working (for other reasons) so there's no backwards compatibility issue if we change it

Perfect.

Quoted 3 lines

Only possible issue is with OCI, and this series makes enough changes that anyone who upgrades kdevops will need to rework their .config anyway.

Makes sense.

Quoted 9 lines

The series has a patch at the end (maybe I didn't post that) that updates docs/kdevops-terraform.md.

On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.

Sorry I failed to understand this, what is mean by separate cloud instances?

Luis

#5 lore raw
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever

On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.

Sorry I failed to understand this, what is mean by separate cloud instances?

The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.

In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.

Signature · 1 line
Chuck Lever
#6 lore raw
Quoted 16 lines · Chuck Lever
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever

On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.

Sorry I failed to understand this, what is mean by separate cloud instances?

The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.

Quoted 4 lines

In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.

I see, so you first bring that cloud instance, and then use that as your command and control for test nodes. Does that first cloud instance need to be instantiated through another kdevops setup?

Luis

#7 lore raw
Quoted 17 lines · Luis Chamberlain
Quoted 16 lines · Chuck Lever
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever

On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.

Sorry I failed to understand this, what is mean by separate cloud instances?

The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.

Quoted 8 lines
Quoted 4 lines

In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.

I see, so you first bring that cloud instance, and then use that as your command and control for test nodes. Does that first cloud instance need to be instantiated through another kdevops setup?

No, you can create it with terraform or from the cloud console. Start it up when you want, or leave it running.

I'm wondering if we can get a cloud devops pipeline to trigger it when it sees a PR against a watched repo. Haven't really thought that through.

Signature · 1 line
Chuck Lever
#8 lore raw
Quoted 27 lines · Chuck Lever
Quoted 17 lines · Luis Chamberlain
Quoted 16 lines · Chuck Lever
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever

On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.

Sorry I failed to understand this, what is mean by separate cloud instances?

The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.

Quoted 8 lines
Quoted 4 lines

In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.

I see, so you first bring that cloud instance, and then use that as your command and control for test nodes. Does that first cloud instance need to be instantiated through another kdevops setup?

Quoted 6 lines

No, you can create it with terraform or from the cloud console. Start it up when you want, or leave it running.

I'm wondering if we can get a cloud devops pipeline to trigger it when it sees a PR against a watched repo. Haven't really thought that through.

OK well so when we add a new commit to kdevops we should ideally try to run a test if the commit is related to terraform or all commits to just verify we don't break cloud instance support. Given we already rely on dedicated github hosted servers, if such hosted servers have the credential files in place then the below would work to automatically trigger a cloud instance test.

For example below could be a .github/workflows/aws.yml which would run on every single commit we submit to kdevops. This would just do a bringup and destroy.

If the console creates a cloud instance, and github can get access to it as a dedicated self hosted runner, and it will persist then that guest can simply just be a bare bones guest to run ansible as the command and control center. And the example below could be used to push github actions onto the runner.

Could this step be removed and have github directly do the bringup? I suspect we can rely on github docker containers which just run kdevops configs for the cloud for it, but they won't be hosted on the respective cloud of choice. Other than that, it would be nice if github integrated support for instantiating a simple guest just to be command control. Otherwise a dedicated self hosted runner outside of the cloud could just have the credentials file.

The kdeops-ci repo [0] has sufficient workflow branch examples, some of which we are already using for different subsystems to trigger runs with kevops on dedicated servers. If a subsystem wanted to leverage the cloud their defconfig would just have the setup for the respective cloud of choice. For example defconfigs/linux-xfs-kpd is what se use for testing pushes to branches to the linux-kdevops/linux-xfs-kpd and only Carlos has access to push to that repo. Which dedicated self hosted runner is opaque to kdevops, but it can easily have cloud credentials to easily do bringup.

I hadn't enabled the below on kdevops just because of lack of time of testing, and trying to then optimize the cost so that a super bare bones cloud istanace would be used for every kdevops commit pushed. However, that's a bit extreme of course, and we can tone it down to only commits perhaps that touch kdevops. But if that works, then by all means so would a push to a git tree as we do with linux-kdevops/linux-xfs-kpd using the examples in the branches for /linux-kdevops/kdevops-ci.

[0] https://github.com/linux-kdevops/kdevops-ci

name: Run kdevops on self-hosted runner for aws cloud instances

on:
workflow_dispatch: # Add this for manual triggering of the workflow

jobs:
run-kdevops:
name: Run kdevops CI
runs-on: [self-hosted, Linux, X64]
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set CI metadata for kdevops-results-archive
run: |
echo "$(basename ${{ github.repository }})" > ci.trigger LOG=$(git log -1 --pretty=format:"%s") > ci.subject echo "cloud aws:" $LOG" > ci.subject
# Start out pessimistic
echo "not ok" > ci.result
echo "Nothing to write home about." > ci.commit_extra

- name: Set kdevops path
run: echo "KDEVOPS_PATH=$GITHUB_WORKSPACE" >> $GITHUB_ENV

- name: Configure git
run: |
git config --global --add safe.directory '*' git config --global user.name "kdevops"
git config --global user.email "[email protected]"

- name: Run kdevops make defconfig-repo
run: |
KDEVOPS_TREE_REF="${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}" SHORT_PREFIX="$(echo ${KDEVOPS_TREE_REF:0:12})"
make KDEVOPS_HOSTS_PREFIX="$SHORT_PREFIX" defconfig-aws

- name: Run kdevops make
run: |
make -j$(nproc)

- name: Run kdevops make bringup
run: |
make bringup

- name: Just check the kernel
run: |
make uname
echo "ok" > ci.result

- name: Start SSH Agent
if: always() # Ensure this step runs even if previous steps failed
uses: webfactory/[email protected]
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

- name: Build our kdevops archive results
if: always() # This ensures the step runs even if previous steps failed
run: |
make ci-archive

- name: Upload our kdevops results archive
if: always() # This ensures the step runs even if previous steps failed
uses: actions/upload-artifact@v4
with:
name: kdevops-ci-results
path: ${{ env.KDEVOPS_PATH }}/archive/*.zip

# Ensure make destroy always runs, even on failure
- name: Run kdevops make destroy
if: always() # This ensures the step runs even if previous steps failed
run: |
make destroy

#9 lore raw
Quoted 46 lines · Luis Chamberlain
Quoted 27 lines · Chuck Lever
Quoted 17 lines · Luis Chamberlain
Quoted 16 lines · Chuck Lever
Quoted 9 lines · Luis Chamberlain
Quoted 5 lines · Chuck Lever

On a related topic, if we ever want to fully support running the kdevops /control host/ in the cloud, terraform supports an authentication mechanism that just uses the local instance's service principal, so no extra authentication material is needed for provisioning the test runners as separate cloud instances. Interesting to consider.

Sorry I failed to understand this, what is mean by separate cloud instances?

The usual situation is the control host (where terraform runs) is outside the cloud. Like, on your workstation. I'm talking about a scenario where kdevops and terraform are running on an instance in the same cloud as where your target nodes are going to run.

Quoted 8 lines
Quoted 4 lines

In that case, terraform can scoop up the service principal for that instance, and use it in place of dot file authentication parameters. So you don't have to maintain the dot file on the instance that is running terraform, if it is already in the cloud.

I see, so you first bring that cloud instance, and then use that as your command and control for test nodes. Does that first cloud instance need to be instantiated through another kdevops setup?

Quoted 6 lines

No, you can create it with terraform or from the cloud console. Start it up when you want, or leave it running.

I'm wondering if we can get a cloud devops pipeline to trigger it when it sees a PR against a watched repo. Haven't really thought that through.

OK well so when we add a new commit to kdevops we should ideally try to run a test if the commit is related to terraform or all commits to just verify we don't break cloud instance support. Given we already rely on dedicated github hosted servers, if such hosted servers have the credential files in place then the below would work to automatically trigger a cloud instance test.

For example below could be a .github/workflows/aws.yml which would run on every single commit we submit to kdevops. This would just do a bringup and destroy.

Quoted 7 lines

If the console creates a cloud instance, and github can get access to it as a dedicated self hosted runner, and it will persist then that guest can simply just be a bare bones guest to run ansible as the command and control center. And the example below could be used to push github actions onto the runner.

Could this step be removed and have github directly do the bringup?

I think you can hide the authentication material in one or more GitHub secrets, if we were to explicitly enable support for passing terraform authentication parameters via environment variables.

Then the GH Action can just run the terraform steps ("make bringup" and "make destroy") without effort. But more below.

Quoted 6 lines

I suspect we can rely on github docker containers which just run kdevops configs for the cloud for it, but they won't be hosted on the respective cloud of choice. Other than that, it would be nice if github integrated support for instantiating a simple guest just to be command control. Otherwise a dedicated self hosted runner outside of the cloud could just have the credentials file.

As far as I can tell, the major four all implement devops pipelines that can hook into GitHub pull requests and bring up instances and what not. Seems like that might obviate the need for managing authentication material on GitHub itself...?

Check out AWS CloudBuild:

https://docs.aws.amazon.com/codebuild/latest/userguide/concepts.html

You give it a little yaml file that tells it to watch a GitHub repo and then execute steps on AWS when it notices a PR. Very much like a GitHub Action, but the runner is on AWS and not GitHub, and because the activity is triggered by an AWS service (CloudBuild) the API authentication step is already done.

OCI calls this "Instance Principal Authentication":

https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm#sdk_authentication_methods_instance_principaldita

Basically the service principal in the instance where terraform is running is used to provision new terraform resources.

So we set up one of these CloudBuild projects for each cloud provider, and for any PR in linux-kdevops/kdevops, it will try a bringup/destroy sequence on that cloud. The build logs can be placed in a public spot on the AWS web site.

</handwave>

So I would need to add a Kconfig widget to each cloud provider's submenu to select "Instance Principal Authentication" (or whatever terminology that cloud provider users) and then kdevops will set up the terraform provider block properly for that scenario.

Quoted 10 lines

The kdeops-ci repo [0] has sufficient workflow branch examples, some of which we are already using for different subsystems to trigger runs with kevops on dedicated servers. If a subsystem wanted to leverage the cloud their defconfig would just have the setup for the respective cloud of choice. For example defconfigs/linux-xfs-kpd is what se use for testing pushes to branches to the linux-kdevops/linux-xfs-kpd and only Carlos has access to push to that repo. Which dedicated self hosted runner is opaque to kdevops, but it can easily have cloud credentials to easily do bringup.

Quoted 3 lines

I hadn't enabled the below on kdevops just because of lack of time of testing, and trying to then optimize the cost so that a super bare bones cloud istanace would be used for every kdevops commit pushed.

As long as the instance configurations are fairly minimal, they can be kept in the free tier on most if not all cloud providers. You do need an active credit card to open a cloud account, though.

Quoted 81 lines

However,
that's a bit extreme of course, and we can tone it down to only commits perhaps that touch kdevops. But if that works, then by all means so would a push to a git tree as we do with linux-kdevops/linux-xfs-kpd using the examples in the branches for /linux-kdevops/kdevops-ci.

[0] https://github.com/linux-kdevops/kdevops-ci

name: Run kdevops on self-hosted runner for aws cloud instances

on:
workflow_dispatch: # Add this for manual triggering of the workflow

jobs:
run-kdevops:
name: Run kdevops CI
runs-on: [self-hosted, Linux, X64]
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set CI metadata for kdevops-results-archive
run: |
echo "$(basename ${{ github.repository }})" > ci.trigger LOG=$(git log -1 --pretty=format:"%s") > ci.subject echo "cloud aws:" $LOG" > ci.subject
# Start out pessimistic
echo "not ok" > ci.result
echo "Nothing to write home about." > ci.commit_extra

- name: Set kdevops path
run: echo "KDEVOPS_PATH=$GITHUB_WORKSPACE" >> $GITHUB_ENV

- name: Configure git
run: |
git config --global --add safe.directory '*' git config --global user.name "kdevops"
git config --global user.email "[email protected]"

- name: Run kdevops make defconfig-repo
run: |
KDEVOPS_TREE_REF="${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}" SHORT_PREFIX="$(echo ${KDEVOPS_TREE_REF:0:12})"
make KDEVOPS_HOSTS_PREFIX="$SHORT_PREFIX" defconfig-aws

- name: Run kdevops make
run: |
make -j$(nproc)

- name: Run kdevops make bringup
run: |
make bringup

- name: Just check the kernel
run: |
make uname
echo "ok" > ci.result

- name: Start SSH Agent
if: always() # Ensure this step runs even if previous steps failed
uses: webfactory/[email protected]
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}

- name: Build our kdevops archive results
if: always() # This ensures the step runs even if previous steps failed
run: |
make ci-archive

- name: Upload our kdevops results archive
if: always() # This ensures the step runs even if previous steps failed
uses: actions/upload-artifact@v4
with:
name: kdevops-ci-results
path: ${{ env.KDEVOPS_PATH }}/archive/*.zip

# Ensure make destroy always runs, even on failure
- name: Run kdevops make destroy
if: always() # This ensures the step runs even if previous steps failed
run: |
make destroy

Signature · 1 line
Chuck Lever
Reviewed-by on v1 1/1
#10 lore raw
In reply to [email protected]
Quoted 6 lines · On Thu, Apr 03, 2025 at 10:49:06AM -0400, [email protected] wrote:

From: Chuck Lever <[email protected]>

Instead of storing authentication secrets in the kdevops .config file, pull them from the authentication profiles already set up in ~/.oci/config. This arrangement is more secure.

Quoted 6 lines

terraform's API authentication is now managed outside of Kconfig, as is done with AWS. An update to docs/kdevops-terraform.md to follow.

Awesome! Might be useful to suggest this can be also be an enabler to use cloud for CIs and later share .config through kdevops-results-archive so to enable to reproduce tests on the cloud easily without revealing cloud secrets.

Luis