Terragrunt is a thin wrapper that provides extra tools for keeping your Terraform configurations DRY. Over the years, as we added new constructs to Terragrunt (e.g. dependency blocks), we realized that some of these new constructs accidentally resulted in duplication within Terragrunt configurations, breaking the promise of DRY.
To address this, we released a stream of new updates to Terragrunt that introduce several different ways to reuse code in Terragrunt. Using these features, we were able to dramatically reduce duplication in the Terragrunt code in the Gruntwork Reference Architecture, cutting down the footprint by 48.5% (NOTE: link is for Gruntwork subscribers only)!
In this blog post, I’ll give an overview of some of these features and how you can leverage them drastically reduce your code footprint:
We’ll conclude the post with general guidelines on how you can update your existing projects to leverage these features, including a link to a a step by step guide.
NOTE: This blog post assumes you already use Terragrunt and have Terragrunt version v0.35.0 or newer.
In our previous post, we focused primarily on how you can make your Terraform code DRY and maintainable with Terragrunt. The settings that needed to be DRY in that post was focused on project level settings which are consistent for the whole project. Things like the remote state backend or Terraform arguments do not change much from environment to environment.
To support this, Terragrunt provides include blocks which allow you to import common settings. Before v0.32.0, Terragrunt only supported a single include block per configuration, and a single level of includes. This meant that you could only import a single set of common settings into your project. However, oftentimes you have many common configuration variables at the component level across environments.
Let’s take a look at an example. Suppose you have a generic VPC module, and you want to always configure VPC flow logs. This might manifest itself as:
terragrunt.hcl
terraform {
source =
"github.com:foo/infrastructure-modules.git//vpc?ref=v0.1.0"
}include {
path = find_in_parent_folders()
}dependency "mgmt" {
config_path = "${get_terragrunt_dir()}/../vpc-mgmt"
}inputs = {
name = "main"
enable_flow_logs = true
flow_logs_cloudwatch_log_group_name = "main-vpc-flow-logs"peer_from_vpc = dependency.mgmt.outputs.vpc_id
}In a traditional Terragrunt folder structure, this configuration will live for each account and region that needs this VPC. That is, this file will be duplicated in every terragrunt.hcl file referenced in the following folder tree:
└── live
├── terragrunt.hcl
├── prod
│ └── vpc
│ └── terragrunt.hcl
├── qa
│ └── vpc
│ └── terragrunt.hcl
└── stage
└── vpc
└── terragrunt.hclWith only a single level of includes and single block of includes, it is very hard to refactor this in a sane way to ensure that you avoid repeating yourself. To address this, you can use multiple include blocks starting with Terragrunt v0.32.0.
With multiple include blocks support, you can move all the common configuration for the component (VPC in our example) to a single configuration that is imported by all the children.
In our example, we can create a file _envcommon/vpc.hcl that is adjacent to the root terragrunt config with project level settings that contains the VPC config:
terraform {
source =
"github.com:foo/infrastructure-modules.git//vpc?ref=v0.1.0"
}dependency "mgmt" {
config_path = "${get_terragrunt_dir()}/../vpc-mgmt"
}inputs = {
name = "main"
enable_flow_logs = true
flow_logs_cloudwatch_log_group_name = "main-vpc-flow-logs"peer_from_vpc = dependency.mgmt.outputs.vpc_id
}The folder structure now looks as follows:
└── live
├── terragrunt.hcl
├── _envcommon
│ └── vpc.hcl
├── prod
│ └── vpc
│ └── terragrunt.hcl
├── qa
│ └── vpc
│ └── terragrunt.hcl
└── stage
└── vpc
└── terragrunt.hclNow, the terragrunt.hcl in each of the accounts becomes the following instead of what we had before:
include "root" {
path = find_in_parent_folders()
}include "envcommon" {
# To locate the envcommon config file, we use the root
# terragrunt config file as an anchor.
path = "${dirname(find_in_parent_folders())}/_envcommon/vpc.hcl"
}If you have a ton of duplicated configuration variables across environments for your component, this can greatly reduce your code footprint in each child configuration.
Multiple include blocks are helpful for duplicating top level configuration variables, but it can be tricky to DRY nested attributes. By default Terragrunt shallow merges the included configuration, which means that keys that overlap are replaced instead of recursively merged.
For example, suppose you have an ECS Service module that takes the container image as an object input of the form:
variable "container_image" {
type = object({
# Repository of the docker image
# (e.g. gruntwork/frontend-service)
repository = string# The tag of the docker image to deploy.
tag = string# The image pull policy. Can be one of
# IfNotPresent, Always, or Never.
pull_policy = string
})
}It is typical for only the tag attribute to be different across environments. With shallow merge, you would have to duplicate this object across the environments even if tag is the only attribute that changes. This is because the container_image object in the parent can only be merged with replacement.
If you had the following in the parent configuration:
inputs = {
container_image = {
repository = "gruntwork-frontend-service"
tag = "v0.0.4"
pull_policy = "IfNotPresent"
}
}The only way to update the tag in the child configuration is by duplicating the common keys in the child:
inputs = {
# You must specify the other keys even if the tag is the only
# thing that differs because Terragrunt will replace the object in
# its entirety.
container_image = {
repository = "gruntwork-frontend-service"
tag = "v0.0.5"
pull_policy = "IfNotPresent"
}
}To handle this use case, Terragrunt supports deep merging included configuration files. An included configuration can be deep merged into the current configuration when the merge_strategy attribute is set to "deep". During a deep merge, the following happens:
string and number), the child overrides the parent.This allows you to define common settings for a complex input variable in the common component configuration, and have the child only inject or override a subset of the attributes.
Going back to our container_image example, with deep merge, you can promote the repository and pull_policy attributes to the common configuration:
parent configuration
inputs = {
container_image = {
repository = "gruntwork/aws-sample-app"
pull_policy = "IfNotPresent"
}
}dev configuration
include "parent" {
path = "/path/to/parent/configuration"
merge_strategy = "deep"
}inputs = {
container_image = {
tag = "v0.0.4"
}
}stage configuration
include "parent" {
path = "/path/to/parent/configuration"
merge_strategy = "deep"
}inputs = {
container_image = {
tag = "v0.0.3"
}
}In this way, you can leverage deep merge to refactor complex nested inputs in your Terragrunt configuration to further DRY up the config.
In the previous section, we covered using include and deep merge to DRY common component configurations. While powerful, include has a limitation where the included configuration is statically merged into the child configuration.
In our example, note that the _envcommon/vpc.hcl file hardcodes the vpc module version to v0.1.0 (relevant section pasted below for convenience):
terraform {
source =
"github.com:foo/infrastructure-modules.git//vpc?ref=v0.1.0"
}# ... other blocks omitted for brevity ...What if we want to deploy a different version for each environment? One way you can do this is by redefining the terraform block in the child config. For example, if you want to deploy v0.2.0 in the qa environment, you can do the following:
include "root" {
path = find_in_parent_folders()
}include "envcommon" {
path = "${dirname(find_in_parent_folders())}/_envcommon/vpc.hcl"
}# Override the terraform.source attribute to v0.2.0
terraform {
source =
"github.com:foo/infrastructure-modules.git//vpc?ref=v0.2.0"
}While this works, we now have duplicated the source URL. To avoid repeating the source URL, we can use exposed includes to reference data defined in the parent configurations. To do this, we will refactor our parent configuration to expose the source URL as a local variable instead of defining it into the terraform block:
locals {
source_base_url = "github.com:foo/infrastructure-modules.git//vpc"
}# ... other blocks and attributes omitted for brevity ...We then set the expose attribute to true on the include block in the child configuration so that we can reference the defined data in the parent configuration. Using that, we can construct the terraform source URL without having to repeat the module source:
include "root" {
path = find_in_parent_folders()
}include "envcommon" {
path = "${dirname(find_in_parent_folders())}/_envcommon/vpc.hcl"
expose = true
}# Construct the terraform.source attribute using the
# source_base_url and custom version v0.2.0
terraform {
source = "${include.envcommon.locals.source_base_url}?ref=v0.2.0"
}Note that the availability of values is subject to the configuration parsing order of Terragrunt. This means that you won’t be able to reference later stage values in early stage blocks, like accessing parent inputs in locals.
You can work around some of this limitation by packing values in inputs. Terragrunt passes inputs to Terraform in a way that Terraform ignores input values that do not correspond to an existing variable in the module. For example, if you want to expose a reference variable that uses dependency blocks, you can create a private input value in the parent configuration that references the dependency, and access it using an exposed include:
parent configuration
dependency "vpc" {
config_path = "../vpc"
}inputs = {
# This input variable is not defined in the underlying Terraform
# module. We use _ to decrease the likelihood of
# accidentally using a defined variable here.
_vpc_id = dependency.vpc.outputs.vpc_id
}child configuration
include "parent" {
path = "/path/to/parent/configuration"
expose = true
}inputs = {
network_configuration = {
vpc_id = include.parent.inputs._vpc_id
}
}In this post, we covered some of the recently introduced features of Terragrunt that helps keep your Terragrunt configuration more maintainable. However, it may not be obvious where to start to leverage these features in your project. This may be especially true for Gruntwork customers that currently have a live Reference Architecture.
To help make this transition easier, we published a step by step guide that you can use to identify the common configurations in your Terragrunt project. In the guide, you will find a repeatable list of steps you can take to incrementally refactor your Terragrunt configurations on a component by component basis.
Give the guide a shot to start making your Terragrunt project DRY, and let us know how it works out for you!
NOTE: Although this guide is optimized for the Gruntwork Reference Architecture, the steps are adaptable to any Terragrunt project looking to leverage multiple includes, deep merge, and exposed includes.
Your entire infrastructure. Defined as code. In about a day. Gruntwork.io.