Mydex CIC explains their journey to migrate their infrastructure-as-code from Terraform to OpenTofu, and why the ’tenv’ tool is a great help to manage different platforms with different Terraform or Tofu tools and versions, simultaneously.

# Introduction

At Mydex CIC, when embarking on a large hosting provider migration in 2016, we took the opportunity to start using Terraform to declare all of our infrastructure ‘as code’.

Terraform is one of a number of modern tools that have completely changed the game in terms of infrastructure cloud management tools. The concepts around it can be summarised loosely as:

• Declaring the desired state of infrastructure (what should exist, and how it should exist)
• Abstracting those definitions through the use of providers such that it becomes easier to use a common language syntax to define infrastructure that might exist at different hosting providers. The abstraction of different providers means that interacting with those hosting providers’ APIs becomes invisible to the engineer executing Terraform.
• Making that declaration of desired state something that can exist in regular, flat files which can therefore also be stored in a version control system (thus creating an audit trail of change, and ability to revert that change)
• Comparing what exists already with what’s defined in code, and using the difference between the two to automate either creating what should exist but doesn’t, or removewhat still exists but is no longer declared, without the manual labour it would otherwise take (and the human error it could introduce).

When implementing Terraform, we adopted some really good ideas published by others from day one, such as separating ’environments’ to have their own state files to avoid accidents with production when working with a lower environment. We also separate our platforms across different hosting accounts entirely, to further isolate them in terms of access/permission boundaries, both to reduce the risk of mistakes (quality controls) as well as to help mitigate any ability of malicious actors to ‘pivot’ across platforms (security controls).

Terraform is used by our infrastructure engineers to drive change in the platform. It is also tightly integrated into our deployment workflow for releasing actual application changes, as the container orchestration that forms part of that workflow is defined through Terraform as well.

Broadly, we love Terraform - certainly the concepts around it - and it really changed the game for us, both in terms of change management and as the driver for our shift to using container orchestration for our platforms.

In August 2023, Hashicorp, who created Terraform, announced changes to the licensing model. This caused a lot of concern in the open source community and I.T industry in general. Many considered that the licence change could be hostile to those wishing to continue using the tool.

Fortunately, the community (and the Linux Foundation) got behind a proposed effort to fork the tool and its provider ecosystem, resulting in an alternative called OpenTofu, developed under the MPL 2.0 licence.

The OpenTofu community also created a manifesto explaining its aims, and called for supporters to pledge to help the project in any number of ways. Mydex CIC is proud to be one of many who signed the pledge.

OpenTofu is designed to be a drop-in replacement for Terraform, and that switch is intended to be easy (spoiler: it really was easy, for us!). However, for such a crucial component in our platform architecture, we wanted to be very careful in how we progressed the change.

This blog post will detail our migration process, including the tools we used, our step-by-step migration plan, rollback procedures, and how we managed the migration schedule.

# Adopting an adjacent tool: tenv

To ensure consistency across different platforms, our CD infrastructure and our sysadmins’ workstations, we first took the opportunity to start using tenv.

tenv is a terrific abstraction tool that allows us to control the specific versions of OpenTofu used across our various isolated platforms and environments. This facilitates a smoother upgrade process by letting us ‘phase in’ a new version in an easy way.

## Why tenv?

tenv simplifies version management by providing a consistent way to handle different versions of OpenTofu (or Terraform). tenv detects what version of the tool is needed in order to run OpenTofu or Terraform commands, based on a metadata version file present on the filesystem.

Simply drop in an .opentofu-version or a .terraform-version file in the directory where your code is, with a version number in the file, and tenv takes care of detecting that version, installing the right version of the tool if you don’t already have it, and making sure that that version of the tool is what is used when you execute a command.

If you have another directory of Terraform/Tofu code that requires a different version, tenv will ‘switch’ to that version when you’re working in that directory.

What we found really cool is that it also includes a proxy command called tf, which acts as an abstraction layer between Terraform and OpenTofu.

This meant we could start using tf immediately, even with our existing Terraform environments, prior to switching to OpenTofu, which simplified our migration plan: our CD pipelines didn’t even need to be changed once we switched to OpenTofu - they just kept using the same ’tf’ commands!

For more details on tenv, you can visit their GitHub page per the link above.

## Installing tenv and OpenTofu

Terraform and OpenTofu define ‘infrastructure’ as code. Principally they are about building actual compute, database, storage, and network devices and ensuring they are configured with the right parameters.

As for what runs inside compute instances (e.g servers) - basically, what they should do after they’ve been provisioned with Terraform/OpenTofu - Mydex uses Ansible. This Ansible configuration management is also managed in version control, ensuring we have change management, audit trail and peer review capabilities at that level.

To use tenv on our Jenkins servers (which run what we call our ‘Zero Touch’ continuous deployment pipeline - something we will talk about at another time), we created an Ansible role to manage its installation.

This setup ensures all our platforms and environments install and utilise OpenTofu through tenv in an identical fashion, providing a unified and consistent approach.

# The Migration Plan

Once tenv was in place and our scripts were using it (with the existing Terraform systems), we moved on to actually planning our OpenTofu migration.

Our migration plan was meticulous, based on lessons learned from previous upgrades of Terraform to new versions in the past. It involved several critical steps to ensure a smooth transition.

Here’s a detailed breakdown of how we did it:

## Pre-flight Checklist

• Engineer logs into the AWS account for the platform/environment they are going to migrate to OpenTofu.
• Engineer navigates to S3 and locates the Terraform remote state for each environment. Each environment has a separate bucket for better isolation.
• Engineer double-checks visually that Bucket Versioning is Enabled. Using Versioning on S3 buckets that hold Terraform state is a terrific idea, because the version system then creates a ‘rollback’ option should we need to back out the change. This is true not just for handling migrations, but for any Terraform/OpenTofu ‘apply’ command - bugs or mistakes could mess up the state or the infrastructure, and prior state can help get you out of a jam.
• Engineer documents the Current State: we noted down the date or hash of the current version of the terraform.tfstate file in each bucket, just to make it easier to return to for rollback purposes later. This also stems from a general lesson learned when planning a big task: always write a rollback plan that simplifies your actions as much as possible, as it is harder to think clearly when in a stressed state should something go wrong.

## Migration steps

• Engineer to install tenv in their workstation, like what runs on the Jenkins servers.
• Set OpenTofu version: Ensure the .opentofu-version file exists in each platform’s Terraform ‘parent’ folder, instead of .terraform-version. Our infrastructure repository is structured in this fashion:

terraform/
  	someplatform/
		terraform/
			some-env/
			some-other-env/
  	someotherplatform/
		terraform/
			some-env/
			some-other-env/

In this case, we put the .opentofu-version file in the ’terraform’ directory above ‘some-env’. Technically, we could even have put the .opentofu-version file in the specific environment folder, if we wanted to phase in OpenTofu per-environment, per-platform. However, in our case it made sense to migrate an entire ‘platform’ (and all its environments) in one go.

For the migration, we would remove the .terraform-version file that exists, and replace it with the .opentofu-version.

• Initialise the environment: tf init. This fetches the provider and plugins from the OpenTofu repository system. As part of this change, the ’lock file’ was updated to reflect the hashes of the OpenTofu providers/plugins.
• Plan the changes: tf plan. This is a ‘dry run’ execution to make sure there are no errors. We also expect to see no actual changes, since we aren’t changing our infrastructure - just the tool we use to manage it with.
• Apply the changes: tf apply. Even though there should be no changes to actually apply, this step is very important, as the remote ‘state’ data will silently be rewritten to reflect that it uses OpenTofu now.

If there were no warnings or errors at any step, we proceeded to the next environment following the same steps.

We then committed the change to our infrastructure git repository to contain the updated .opentofu-version file and the updated lock files.

# Rollback Plan

Despite meticulous planning, issues can arise during migration. A rollback plan ensures we can revert to a stable state if needed.

## Rollback Steps

• Engineer would log into the AWS account for the platform being working on.
• Navigate to the S3 buckets and locate the Terraform remote state for each environment.
• Find the Previous State:
  • Go to the Objects tab.
  • Click on terraform.tfstate.
  • Navigate to Versions.
  • Download a previously working version to your machine.
• Restore the Previous State:
  • Upload the downloaded file back to the bucket using the Upload button in the Objects tab.
• Revert to Terraform:
  • Rename the .opentofu-version file to .terraform-version in each platform’s Terraform root folder.
  • Change the version number in that file to (Terraform) 1.5.6.

# Conclusion

We had a very clean, trouble-free migration from Terraform to OpenTofu without any errors, and are delighted to be using a community-friendly infrastructure management tool.

It was another great example of why intricate planning of both the migration itself, as well as a rollback plan, pays off. Our migration was slow (across multiple platforms over two months). We could have moved faster, especially as it became evident that the process was going smoothly. However, we stubbornly stuck to our schedule, because we believe changes like this should appear ‘boring’ and invisible to the rest of our organisation and those we serve. When infrastructure management is ’exciting’, it is often for the wrong reasons. We do not ‘move fast and break things’ at Mydex, however fashionable it may be.

Regarding the licensing issue: we believe the open source ecosystem has a ‘people problem’, in terms of ensuring adequate support is given to those who are often volunteers working on projects that many, both professional and hobbyist, depend heavily on.

Increasingly, attacks on the software supply chain show where this ‘human element’ breaks down, either due to projects/accounts in software repositories being abandoned and taken over by malicious actors, or through coercion of the original developer.

Rather than see free tools get swallowed up by corporations using unfriendly licences - even if this addresses a ‘resource’ issue - we think that the community (and community-friendly organisations, such as the Linux Foundation) rallying around free software is a much healthier way to combat this problem, to ensure those who enjoy working on it feel supported. The OpenTofu manifesto and pledge system was a great example of seeing those who rely on this sort of tool, want to see it succeed for the long term.

Last month, OpenTofu 1.7.0 was released with innovative new features such as state encryption. In this way, we already begin to see the tool improve beyond its roots and are very excited to make use of such new features.

We look forward to fulfilling our pledge and helping OpenTofu through testing of new versions, aiding open source efforts such as issue triage, and so on, wherever and whenever we can do so.

In the meantime, it is also interesting to watch the industry respond to IBM’s acquisition of Hashicorp and what that might mean for Terraform. Was the licence change made in anticipation of that acquisition, or will IBM return Terraform to its original licence, and the projects merge? We doubt the latter, but will be watching.