Best practices for managing Terraform state at scale in modern IaC workflows

Terraform state is information about the infrastructure Terraform has created in your cloud accounts. Terraform uses the state next time it's run to work out which infrastructure changes are required. It's therefore crucial that state files are carefully managed to prevent conflicts, errors, and security issues.

‍

In this article, we're going to explain what to include in a Terraform state management strategy. We'll discuss why Terraform state is so important and share best practices that'll keep your IaC processes running smoothly. Let's jump in.

‍

What is Terraform State Management?

‍

Terraform state management is the process of properly storing and maintaining Terraform's state files. These files, usually suffixed .tfstate, are generated each time you make changes to your infrastructure using terraform apply. They describe the state of the infrastructure in your cloud accounts, allowing Terraform to track the resources it's controlling.

‍

Corrupt or missing state files severely impact IaC workflows. They prevent Terraform from identifying existing resources, potentially causing new changes to replace or destroy vital infrastructure components. Similarly, dangerous state conflicts can occur if two developers run terraform apply simultaneously, potentially causing one developer's state file changes to be overwritten.

‍

Smaller teams won't necessarily encounter these issues, but they quickly become big problems at scale. You can address the challenges by implementing a state management strategy that acknowledges the critical importance of your tfstate files.

‍

Best practices for scalable Terraform state management

‍

Scalable Terraform state management strategies depend on several processes working in unison. You need to plan where you'll store your state files, the ways in which you'll secure them, and how you'll defend against the risk of conflicts posed by multiple terraform apply operations running concurrently. Let's look at the key best practices to follow for safe and reliable Terraform state operations.

‍

Use a remote state backend

‍

Terraform state backends are responsible for storing your state files. By default, Terraform uses the local backend to store your state in a terraform.tfstate file within your project's filesystem. This isn't scalable because the state's only available in the environment where terraform apply was run.

‍

Remote state backends store your state on an external service such as a file server, object storage provider, or web API. This ensures your state files remain accessible across machines and environments. You can safely run terraform apply in CI/CD jobs, for instance, as each job will be able to access the state created by the previous run.

‍

Available state backends include popular storage services and databases such as AWS S3, Google Cloud Storage, Consul, and Postgres. They're configured using a backend block within your Terraform root module. The properties required depend on the backend you're using. Here's a simplistic example of how to store your state files in an S3 bucket using the s3 backend:

‍

terraform {

backend "s3" {

bucket = "terraform-state-file"

region = "us-east-1"

# retrieves credentials from a named profile in an AWS config file

profile = "terraform"

}

}

‍

Adopt IaC automation platforms with integrated Terraform State Management

‍

Properly configuring a Terraform state backend can be complex. You might feel like you already need Terraform to provision your backend's database or object storage bucket!

‍

You can simplify your Terraform state management process by adopting a dedicated infrastructure automation platform such as Spacelift or Env0. These solutions are designed to orchestrate your IaC tools, including both Terraform and alternatives like Pulumi and Ansible. They connect directly to your IaC repositories, then automatically run terraform apply each time changes are made.

‍

Spacelift and Env0 both offer integrated Terraform state backends that you can use for your state files. This eliminates the need to configure and maintain external services while giving you all the benefits of a remote state backend. Spacelift's implementation even lets you roll back to a previous state file version with a single button click.

‍

Enable and use state file locking

‍

State files should be locked each time you run terraform apply. Locked files can't be modified until the lock is released. This prevents conflicts if two developers (or CI/CD jobs) try to run terraform apply at the same time. The second instance won't be able to run until the first one finishes, ensuring it operates on the latest version of the state files.

‍

Responsibility for implementing locking lies with Terraform's state backends. Not all backends support locking, so you should check you're using a remote backend that includes it. Some backends require you to opt-in to locking by setting a config property. For example, you must set use_lockfile to true when you're using the S3 backend:

‍

terraform {

backend "s3" {

bucket = "terraform-state-file"

use_lockfile = true

}

}

‍

Version your state files

‍

Versioning your state files allows you to inspect the changes made to your live infrastructure over time. It also gives you a recovery mechanism in case your state files are accidentally modified or become corrupted.

‍

Integrated platforms like Spacelift help simplify version management by providing a built-in interface to view and restore previous versions. Where applicable, you can also use the versioning features directly available in your state’s file backend. For instance, enabling S3 bucket versioning will ensure older versions of state files can be recovered from AWS. It's also possible to version state files using Git via the terraform-backend-git community project.

‍

Encrypt state files at rest and in transit

‍

Terraform state files precisely document your live infrastructure's configuration. Because they can contain highly sensitive details, they should be continually encrypted. This ensures attackers won't be able to inspect your state files, even if they successfully breach your storage.

‍

Encryption capabilities are again backend dependent, but are available with the major remote services like AWS S3 and Google Cloud Storage. The latter option also supports custom encryption keys for even stronger protection.

‍

Restrict access to state files and credentials

‍

Terraform state files rarely need to be edited by developers. It's therefore best practice to enforce tight access controls that prevent unauthorized changes being made. If you're only using Terraform within a CI/CD pipeline, then developers are unlikely to need their own credentials to access your state backend.

‍

The precise steps to take will depend on which backend you're using. For object storage buckets, you should ensure public access is disabled, then check for any unexpected IAM users that have permission to access your bucket. Similarly, you should lock down Consul or Postgres databases to minimize the threat of over-privileged access.

‍

Regularly isolate, separate, and refactor state files to support scalability

‍

Using one Terraform state file for your entire project quickly becomes impractical as your infrastructure grows. You should regularly refactor your state files by separating the state configs for different resources into their own isolated files. This simplifies maintenance and helps limit how far errors can spread. It can also improve performance because Terraform won't need to check so many resources when you're changing a specific part of your infrastructure.

‍

You can split up state files by creating new Terraform modules. For instance, you could use a nested directory structure with subdirectories for different types of resources:

‍

- project/

- database/

main.tf

- compute/

main.tf

- storage/

main.tf

‍

Each module would then use its own state file.

‍

Built-in Terraform CLI commands allow you to move existing resources between your state files. For instance, the following command migrates the aws_instance.app resource from an existing terraform.tfstate state file into the project/compute directory:

‍

$ terraform state mv -state terraform.tfstate -state-out project/compute/main.tfstate aws_instance.app aws_instance.app

‍

Move "aws_instance.app" to "aws_instance.app"

‍

Successfully moved 1 object(s).

‍

Terraform also supports multiple isolated workspaces in a single module. This is an ideal way to distinguish between the state files for different environments like development and production. Each workspace has its own state file, but shares the module’s single base Terraform configuration. You can set workspace-specific variables and overrides to apply per-environment customizations. This lets you easily deploy multiple instances of your application, then version their state files independently.

‍

Use the Terraform CLI to create and select workspaces:

‍

$ terraform workspace new production

‍

Created and switched to workspace "production"!

‍

‍

$ terraform workspace select production

‍

Terraform uses the default workspace if you haven't selected a named one.

‍

Use drift detection to reconcile state file issues

‍

Infrastructure drift occurs when your live resources no longer match the state defined in your Terraform config. It often happens silently when developers make changes outside Terraform or unwanted automatic updates occur. Drift poses serious challenges for infrastructure operators because it increases the risk of misconfigurations, security issues, and compliance breaches.

‍

Implementing automated drift detection and resolution processes is the most effective defense against these problems. Your Terraform state files contain all the information about what your infrastructure should look like, so it's good practice to put them to use identifying drift. Regularly comparing your infrastructure's actual state to what's described in your state files lets you efficiently find discrepancies.

‍

It's easiest to do this using the built-in drift management features found in new-gen IaC platforms like Spacelift, Env0, and Terraform Cloud. They fully automate the process of finding and fixing drift by running scheduled jobs that look for changes in your infrastructure.

‍

Configure state backend audit logging to track state file changes

‍

State file versioning, access controls, and encryption still aren't always enough to support the most stringent security and compliance requirements. Enabling change audit logging in compatible state backends allows you track state file activity and conduct precise investigations into changes.

‍

If you're using the S3 backend, then you can use AWS' built-in tools to record the requests made to your state files bucket. Similar audit capabilities are also available in Consul and Google Cloud Storage, among other providers.

‍

Import existing infrastructure resources into your Terraform state

‍

Gradually adopting Terraform can help you solve any workflow problems before you go all-in on migrating your entire infrastructure to IaC. However, this also runs the risk of some components lingering as legacy resources that Terraform doesn't know about. This can lead to outdated and misconfigured infrastructure if the resources are then forgotten.

‍

It's best practice to eventually standardize on using IaC for your entire infrastructure, even though this doesn't mean you have to do everything on day-1. You can gradually import existing resources from your cloud accounts to add them to your Terraform state files. This ensures your state files remain an accurate representation of your infrastructure.

‍

The following HCL snippet provides a simple example of importing an existing AWS EC2 instance with the ID i-abc123:

‍

import {

to = aws_instance.app

id = "i-abc123"

}

‍

resource "aws_instance" "app" {

name = "app"

}

‍

After you run terraform apply, the aws_instance.app resource will be added to your Terraform state file. It'll map to the existing EC2 instance with ID i-abc123 in your AWS account.

‍

Summary

‍

Terraform makes infrastructure operations easy, fast, and repeatable, but its state files must be carefully managed to prevent errors and conflicts. Implementing a remote state backend, robust access controls, and state file locking is a crucial step towards making your IaC workflows more reliable.

‍

As manual Terraform state management can quickly become complex, it's often preferable to use a managed all-in-one solution like Spacelift or Env0. These platforms provide an end-to-end IaC and infrastructure automation experience with integrated Terraform state support. They let you focus on managing your infrastructure, instead of maintaining your Terraform state workflow.

‍

Want to learn more about building scalable IaC? Check out our guide to the expert IaC strategies that enable infrastructure automation success at scale. You can also contact our cloud transformation experts at Semantive to get started building your IaC and Spacelift implementation plan.