Mydex CIC describes how to manage secrets using a versitile opensource tool and make them accessible at runtime.

# Introduction to SOPS: Securely Managing Secrets in Cloud-Native Environments

When working with cloud-native environments, securely managing secrets is crucial. Secrets like API keys, passwords, and other sensitive information must be protected while ensuring they are easily accessible to the right components of the system. Storing secrets in version control unencrypted is also a problem. There can be a temptation to not store the secret in an auditable, trackable place at all, which creates a new problem - that of lack of an audit trail or resilience against loss. This is where SOPS (Secrets OPerationS) comes in. SOPS is an open-source tool designed to help developers and DevOps teams securely manage secrets using encryption. Furthermore, such files can be stored in version control encrypted at rest without a problem.

# What is SOPS?

SOPS was originally created by Mozilla, but has since been absorbed into the Cloud Native Computing Foundation (CNCF). It is a file-based encryption tool that encrypts the values of specific fields in a structured file (like YAML, JSON, or ENV) while leaving the rest of the file in plaintext. This allows the user to safely store encrypted secrets in version control systems like Git without exposing sensitive information. SOPS supports multiple encryption backends, including AWS KMS, GCP KMS, Azure Key Vault, and PGP/GPG keys, providing flexibility in how to manage and protect secrets.

## Key Features of SOPS

• File Type Agnostic: SOPS can handle various file types such as YAML, JSON, INI, and ENV files.
• Partial Encryption: SOPS only encrypts the values within a file, leaving the structure intact.
• Multiple Encryption Backends: Supports AWS KMS, GCP KMS, Azure Key Vault, and PGP/GPG or AGE for encryption.
• Version Control Friendly: Works seamlessly with version control systems like Git, as the file structure is retained in plaintext.
• Integrated with various other infrastructure/configuration as code systems
  • OpenTofu/Terraform: can be used to define the values of attributes thanks to a community-provided provider
  • Ansible: can be used to define inventory values thanks to a community collection
  • Kubernetes: Can be used with Kubernetes to manage secrets as part of a CI/CD pipeline.

# How SOPS Works

At its core, SOPS is a command-line tool that encrypts and decrypts data in structured files. It operates by encrypting only the values of specified fields, preserving the file structure and any non-sensitive data. This makes it easy to review and manage secrets in version control systems, as only the encrypted values are obfuscated.

## Encryption and Decryption Process

1. Encryption: When encrypting a file using SOPS, the tool scans the file for values to encrypt based on its configuration. It then encrypts these values using the chosen encryption backend (e.g., GPG, AWS KMS). The encrypted file still retains the original structure and can be committed to version control.
2. Decryption: Decrypting a file with SOPS is straightforward. The tool reads the encrypted file, identifies the encrypted values, and decrypts them using the appropriate decryption keys. The file is then output in its plaintext form.

## Encryption Backends

SOPS supports several encryption backends, which can be used individually or in combination:

• GPG/PGP: Uses GPG keys for encrypting and decrypting data. This method is popular for its simplicity and local key management.
• AWS KMS: Integrates with AWS Key Management Service, allowing the user to encrypt data using KMS keys. This is ideal for AWS-based environments where they want to leverage IAM roles for key management.
• GCP KMS: Similar to AWS KMS, but for Google Cloud Platform. It allows the user to encrypt and decrypt data using Google Cloud KMS keys.
• Azure Key Vault: Integrates with Azure’s Key Vault service, enabling encryption with Azure-managed keys.

## Configuration File (.sops.yaml)

SOPS uses a configuration file, .sops.yaml, which defines how SOPS should handle the encryption and decryption of files. This file can specify things like the encryption backends to use, key management policies, and rules for which keys should be used to encrypt which files (or which parts of a file should be encrypted). Here is a simple example .sops.yaml configuration file:

creation_rules:
- path_regex: '.*\.(yaml|yml|json)$'
- pgp: 34EF567890ABCD1234EF567890ABCD1234

In this example:

Files with extensions .yaml, .yml, or .json in the current working directory (and all subdirectories) will be encrypted. Encryption backend will use PGP with a specific public key for encrypting or decrypting the contents.

## Installation

Installing SOPS is straightforward and it is available for most platforms. You can download the binary from the SOPS GitHub Releases page and install it according to your system’s requirements.

# Using GPG Keys with SOPS

GPG (GNU Privacy Guard) is one of the most common encryption backends used with SOPS (other options such as AGE are also supported). GPG keys allow the user to encrypt files locally without relying on cloud provider vendor-specific solutions.

## Creating a GPG Key

Create a GPG key using the following command:

gpg --full-generate-key

Follow the prompts to generate the key. Once generated, list the GPG keys using:

gpg --list-keys

## Encrypting a File with GPG

To encrypt a file using SOPS and a GPG key, use the following command:

sops -e --pgp <GPG_KEY_ID> file.yaml

Replace <GPG_KEY_ID> with the GPG key’s fingerprint or email address. The -e flag tells SOPS to encrypt the file. You can also store SOPS_PGP_FP as an environment variable for convenience to avoid having to pass the --pgp argument. SOPS allows the user to encrypt files for multiple recipients. For example, they can use multiple GPG keys or KMS keys, so different team members can decrypt the file using their own keys. This is useful for collaborative environments where multiple people need access to the secrets. To encrypt a file for multiple GPG recipients:

sops -e --pgp <GPG_KEY_1> --pgp <GPG_KEY_2> file.yaml

An easier approach is to append those GPG keys in a comma-delimited list in the ‘creation_rules’ in the .sops.yaml file, and rely on that file detecting the correct GPG keys to use based on the path. Decrypting a File with GPG To decrypt a file encrypted with GPG, simply use:

sops -d file.yaml

SOPS will automatically detect the encryption method and use the GPG key to decrypt the file.

# How Mydex uses SOPS

## Ansible

Integrating SOPS encrypted YAML files in Ansible is as simple as ensuring you have loaded that collection in the ansible.cfg.

vars_plugins_enabled = host_group_vars,community.sops.sops

With this configuration in place, any inventory file with the .sops.yml extension will be decrypted on-the-fly as Ansible runs the playbook. This integration ensures that sensitive data remains encrypted at rest but accessible during runtime.

## Terraform or OpenTofu

To configure SOPS to work with Terraform/OpenTofu is also straightforward. The carlpett/sops provider needs to be configured in the required_providers block:

terraform {
    required_providers {
	sops = {
  	source  = "carlpett/sops"
  	version = "~> 0.5"
	}
    }
}

Then, instantiate the provider and tell it which SOPS file to read from as a data resource:

provider "sops" {}
data "sops_file" "secrets-creds" {
  source_file = "secret_creds.sops.yml"
}

Now any values from the SOPS file can be passed to a module by referencing the secret key of the needed value:

mysecretvalue_variable = data.sops_file.secrets-creds.data.somesecretkey

Note that it is the key referenced in the SOPS file that we are referencing, in order to load its value.

# Features Inside SOPS

SOPS comes with several built-in tools and features that enhance its functionality:

## Editor

Although SOPS can be mistaken for an editor, it’s actually not; rather, it acts as a wrapper around the user’s preferred text editor. When using the sops command to edit a file, it automatically decrypts the file, opens it in the default editor, and re-encrypts it when save and exit.

sops file.yaml

This command opens file.yaml in the text editor. Once saved and closed the editor, SOPS re-encrypts the file.

## JSON and ENV File Support

SOPS supports JSON and ENV files, making it versatile for different use cases.

• JSON Files: the user can encrypt specific fields within a JSON file, leaving the rest in plaintext.
• ENV Files: SOPS can encrypt values in an environment file (.env), which is useful for securely storing environment variables.

## Key Rotation

SOPS supports key rotation, allowing the user to update the encryption keys without re-encrypting all the files manually. This is particularly useful when keys need to be rotated periodically for security reasons. To rotate keys, use the following command:

sops -r --in-place file.yaml

This command decrypts the file with the old keys, re-encrypts it with the new keys defined in the .sops.yaml configuration, and saves it in place. If you change the list of GPG keys in the .sops.yaml file, then sops updatekeys will also automatically rotate the file with the associated encrypted data for that GPG key removed from the file.

## Data Integrity with HMAC

SOPS includes support for HMAC (Hash-based Message Authentication Code) to ensure data integrity. When enabled, SOPS generates an HMAC for the encrypted file, which can be verified during decryption to detect any tampering. Conclusion SOPS is a powerful and flexible tool for securely managing secrets. Whether working with Ansible, OpenTofu, Kubernetes, or other cloud platforms, SOPS provides a secure and convenient way to store and utilise sensitive information. Understanding how SOPS works, configuring it correctly, and utilising its various tools and features, can significantly improve the security and manageability of secrets. With SOPS, secrets can also confidently be stored in version control, knowing they are encrypted and secure, making SOPS an invaluable tool in any DevOps toolkit.