Gruntwork Newsletter, August 2022

Every few months, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made since the last newsletter and news from the DevOps industry. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

Since the last newsletter, we’ve updated all of the code in the IaC Library to work with Terraform AWS Provider v4, released a blog post series on how to manage multiple environments with Terraform, created a crash course series on Docker, Kubernetes, Terraform, and AWS, shared a talk on Cloud Adoption Fails, and wrote up a blog post on how to write code on Mac or Linux but test on Windows with hot-reloading.

As always, if you have any questions or need help, email us at support@gruntwork.io!

• Gruntwork Updates
• DevOps News

Gruntwork Updates

Terraform AWS Provider v4 Support Complete

Motivation: HashiCorp released AWS provider v4 earlier this year, which was a major refactor of several resources, introducing breaking changes. Customers using our modules experienced configuration errors as their modules pulled the latest provider and tried to apply the same configuration. HashiCorp has since reverted all of these breaking changes in 4.9.0 (see the first note in their Upgrade Guide).

Solution: Within a few days of the release we locked down all our modules to be limited to 3.x versions of the provider to stopgap the failures that affected customers. Then we worked to unlock each module repo and service catalog repo, by building in support for 4.x without requiring configuration changes by customers in most cases. We also wanted to make this upgrade safe. Users who upgrade to the new unlocked version will be unlikely to face downtime. We ensured this by running a real upgrade from the latest release to the 4.x unlocked version, on select modules in each module repo. The final repos have been unlocked with support for 4.x as of early August.

What to do about it: Follow this guide to upgrade your module sources to the relevant versions of Gruntwork modules. You only will need to run terraform init -upgrade, followed up with terraform plan and apply. No configuration changes are required if using our modules, but you may need to update your own code, as per HashiCorp’s upgrade guide. Please let us know how it goes!

How to manage multiple environments with Terraform

Motivation: A question we’ve gotten over and over again is, “what’s the right way to manage multiple environments with Terraform?” Workspaces? Branches? Terragrunt?

Solution: We wrote a blog post series called how to manage multiple environments with Terraform that does an in-depth walkthrough and comparison of these three options:

• Part 1: Workspaces
• Part 2: Branches
• Part 3: Terragrunt

What to do about it: Check out the blog post series and let us know what you think!

The Docker, Kubernetes, Terraform, and AWS crash course series.

Motivation: Just about every single Gruntwork customer is interested in using Terraform, Docker, Kubernetes, and AWS. However, not every one of our customers knows how to actually use these technologies, and sometimes, they were afraid to ask or unsure where to start.

Solution: We created the The Docker, Kubernetes, Terraform, and AWS series to help developers overcome the fear of DevOps by going through a series of lightning quick crash courses where you learn by doing. Each course is designed for newbies, starting at zero, and building your mental model step-by-step through simple examples you run on your computer to do something useful with one of these modern DevOps tools — in minutes:

1. A crash course on Docker
2. A crash course on Kubernetes
3. A crash course on AWS
4. A crash course on Terraform

What to do about it: Check out the blog post series and let us know what you think!

Cloud Adoption Fails

Motivation: At Gruntwork, we’ve had the chance to see the cloud adoption journeys of hundreds of companies, from tiny startups to Fortune 50 giants. We’ve seen many of those journeys go well — but we’ve also seen our share of failures, including massive cost overruns, endless death marches, security disasters, and so on.

Solution: To help your cloud journey go well, we looked at the companies that were successful and extracted from them the patterns they had in common, distilling all this experience down into something called the Gruntwork Production Framework, which defines five must-have ingredients to adopt the cloud at your own company — and hopefully, to end up with your very own happy cloud deployment. Today, we’re happy to share the recording of a talk from the 2022 Philly ETE Conference where Gruntwork co-founder Yevgeniy Brikman introduces each ingredient of the Gruntwork Production Framework and how to put them together: Cloud Adoption Fails: 5 Ways Deployments Go Wrong and 5 Solutions.

What to do about it: Check out the talk and let us know what you think!

How to write code on Mac or Linux but test on Windows with hot-reloading

Motivation: Writing code that has to run on multiple platforms is hard, but our blog post and companion code on launching EC2 instances as test servers makes it much easier.

Solution: We have published a new blog post called How to write code on Mac or Linux but test on Windows with hot-reloading, which provides a Packer template and all the Terraform code necessary to follow along and launch your own test Windows instances. Best of all, we’ll show you how to set up “hot-reloading” so you can edit your code on your preferred work machine but test your changes instantaneously on your Windows host.

What to do about it: Have a read through the post and let us know what you think!

Service Catalog Updates

terraform-aws-service-catalog

• v0.90.6: Exposed the ability to add service principals to the Cloudtrail key.
• v0.90.7: Updated public-static-website to support restricting access to S3 Bucket to only CloudFront. This will make the S3 Bucket private instead of public. Note that this is only supported if the private bucket is deployed in us-east-1.
• v0.91.0: Added the ability to restrict IMDS on EKS workers (self managed and managed node groups). Additionally, the launch template is now updated to restrict the service to require tokens by default (IMDS v2).
• v0.92.0: Added support for binding additional TLS/ACM certificates to an ALB, via the new additional_ssl_certs_for_ports input variable. Updated various dependencies — refer to the release notes for more details.
• v0.92.1: Added support for making GuardDuty optional in account-baseline modules via the enable_guardduty variable. This is useful when setting up org level GuardDuty access.
• v0.93.0: Added support for deploying k8s-service in headless mode. Updated default user-data scripts to prefix the filename with _ so that it is run first.
• v0.93.1: Added documentation for a hidden parameter lifecycle_policy_rules for ECR Repos; Added documentation for a hidden parameter taints for Managed Node Groups; Added support for specifying a custom security group name for RDS and ECS Services; Added support for configuring performance insights on Aurora; Updated dependencies — refer to the release notes for more details.
• v0.93.2: Exposed the ability to specify tags on ecs-cluster resources; Updated ARN references to dynamically look up the AWS partition information to support alternative partitions; Updated various dependencies — refer to the release notes for more details.
• v0.94.0: Landing Zone modules only: Unlocked AWS provider v4, requiring minimum 3.75.1 . This update includes a few tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.94.1: Updated aurora module to output the generated security group ID; Updated the website S3 bucket created in the public-static-website module with additional security configurations when operating in private bucket mode; Added support for configuring CloudFront Functions, a more performant and lightweight alternative to Lambda@Edge, with static websites; Added support for implementing default directory indexing for private S3 bucket backed static websites; Added instructions to README on how to perform a blue-green deployment of Aurora.
• v0.94.2: Updated dependency terraform-aws-data-storage from v0.24.0 to v0.24.2; Exposed new parameters to pass through permission boundaries to IAM Roles managed by the account-baseline modules.
• v0.95.0: Updated sns-topics module to require passing through the Slack webhook URL using AWS Secrets Manager instead of directly as module variables; Updated dependencies — refer to the release notes for more details.
• v0.95.1: Exposed the ability to set AWS Tags on the resources managed by the ecs-deploy-runner module. Updated dependencies — refer to the release notes for more details.
• v0.96.0: (Continuing with efforts on v0.94.0.) All remaining modules: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.96.1: We missed a few underlying module updates needed in v0.96.0. This release ensures all underlying modules have the provider v4 support.
• v0.96.2: Exposed the cleanup_on_fail parameter in k8s-service module's helm_release resource; Updated landingzone/account-baseline-root to expose advanced_event_selectors for Cloudtrail as cloudtrail_advanced_event_selectors; Updated rds module to make the option_group_name parameter configurable; Updated jenkins to allow configuring without a Route53 entry; Updated dependency terraform-aws-openvpn(v0.24.1 to v0.24.3).

terraform-aws-cis-service-catalog

• v0.35.1: Updated dependencies — refer to the release notes for more details.
• v0.35.2: Added a new option enable_cis_check (security_hub_enable_cis_check in landingzone modules) which can be used to disable the CIS standards check in AWS SecurityHub. These are enabled by default to preserve current behavior. DIsabling them is useful if you are using another tool to run the checks, such as Steampipe.
• v0.35.3: Fixed bug where CloudWatch could not access the SNS Topic for the log metric filters when they are encrypted with KMS.
• v0.35.4: Exposed the ability to configure max_password_age and password hard_expiry for the IAM Password Policy from account-baseline-security.
• v0.35.5: Added the ability to disable private SSH and RDP access at the NACL level. Added the ability to configure the remote administrative ports for the NACLs from the VPC layer.
• v0.35.6: Added the ability to configure an aggregator region to roll up the findings to in Security Hub. Note that only administrator accounts can designate an aggregator region.
• v0.35.7: Fixed a CIS Compliance misinterpretation for control 5.1 (VPC NACLs). Control 5.1 states Ensure no Network ACLs allow ingress from 0.0.0.0/0 to remote server administration ports, which was originally interpreted to mean that there should be no network ACL that effectively allows ingress from 0.0.0.0/0 on admin ports, but the popular interpretation is that there should be no network ACL rule that allows ingress from 0.0.0.0/0, even if a higher priority rule explicitly denies access. As such, starting this release, the network ACL rules are adjusted to ensure that no rule allows ingress from 0.0.0.0/0 in all rules configured.
• v0.36.0: Updated the AWS Config Aggregator to force to aggregating all regions. This is due to the CIS control 3.5 that requires AWS Config be enabled in all regions, so it makes sense to always require aggregating all regions. Updated dependencies — refer to the release notes for more details.
• v0.36.1: Added the ability to configure a subscription to the SNS Topic for CloudTrail CIS benchmark alarms that sends the alerts to a Slack Channel.
• v0.37.0: Updated cloudtrail to configure object level data logging for ALL S3 buckets, not just the CloudTrail bucket. Previously the compliance requirement 3.10 and 3.11 was misinterpreted to mean only the CloudTrail bucket when in reality all S3 buckets in the account required to be monitored. Exposed the additional_data_logging_resources input variable in the account-baseline modules to configure additional data logging resources on the CloudTrail instance. Added the ability to configure AWS Organizations metric filters in child accounts. While these are not strictly necessary, configuring them may be useful for satisfying various automated checks for CIS. Fixed bug where the CloudTrail metric filter SNS topics were not being encrypted with the CloudTrail KMS Key as originally intended.
• v0.37.1: Exposed the ability to configure cross account SNS Topic access. You can use the new cloudtrail_benchmark_alarm_external_aws_account_ids_with_publish_access and cloudtrail_benchmark_alarm_external_iam_entities_with_list_subscription_access input variables to configure cross account access for various scenarios.
• v0.38.0: Updated hardcoded ARNs to lookup AWS partition information to support deployment into alternative partitions such as GovCloud; Update dependencies — refer to the release notes for more details.
• v0.38.1: Exposed flag enable_root_account_mfa_rule for AWS Config to allow disabling the root user MFA AWS Config rule. This should only be turned off for AWS partitions that do not support a root user (e.g., GovCloud).
• v0.38.2: Updated various dependencies — refer to the release notes for more details.
• v0.39.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.40.0: Renamed variable associate_to_master_account_id to associate_to_admin_account_id in aws-securityhub module to align with latest AWS documentation.
• v0.40.1: Update dependencies — refer to the release notes for more details.
• v0.41.0: In v0.39.0, which aimed to fully unlock the AWS v4 Provider, we missed a few module updates. This release ensures all underlying modules support AWS Provider v4.

Open Source Updates

Terragrunt

• v0.38.5: Fixed a bug in render-json where dependency blocks and locals information were lost in the resulting rendered config. Improved error logging in Terragrunt. Updated dependencies that security patches.

Terratest

• v0.40.19: Fixed a bug in the Clone function of the terraform.Options struct where map objects were not being deep cloned, causing side effects.
• v0.40.20: Exposed DownloadPolicyE function so that it can be called independent of OPA functionality.

cloud-nuke

• v0.12: Implement support for inspecting and nuking GuardDuty detectors
• v0.12.3: Introduce support for passing external credentials via aws.Config when scripting against cloud-nuke as a library
• v0.13.0: Introduce support for inspecting and nuking Macie member accounts
• v0.14.0: Introduce support for inspecting and nuking IAM Roles
• v0.15.0: Introduce support for inspecting and nuking Sagemaker notebook instances
• v0.16.0: Introduce support for inspecting and nuking Kinesis Stream instances
• v0.16.1: Fix VPC nuking behavior by supporting deletion of chained security groups in nukeSecurityGroups
• v0.16.2: Fix Elasticache nuking behavior by implementing support for nuking cache clusters that are members of replication groups.
• v0.16.3: Fix Elasticache nuking behavior by looking up replication groups as well as single cache clusters in our list method.
• v0.16.4: Fixed bug where global resources were not nuked due to a broken security token issue when using GovCloud.
• v0.17.0: Introduce support for inspecting and nuking API Gateways (v1 and v2)
• v0.18.0: Introduce support for inspecting and nuking Elastic FileSystems (EFS)
• v0.19.0: Introduce support for inspecting and nuking SNS Topics

git-xargs

• v0.1.0: Prettify reporting with pterm.sh. Append [skip ci] to commits by default and provide the --no-skip-ci flag. Fix lingering concurrency issues.

gruntwork

• v0.3.8: Add new command gruntwork vault generate that assists you in generating valid aws-vault account profiles for your Ref Arch AWS accounts, to ease login and executing commands.
• v0.3.9: Add new commands: gruntwork vault login and gruntwork vault exec to make it easier to quickly log in or execute arbitrary commands against any of the AWS accounts in your Reference Architecture. Designed to be used in conjunction with gruntwork vault generate
• v0.3.10: Fix a bug causing the git ref used to fetch the boilerplate template for the form wizard to be out of date.

helm-kubernetes-services

• v0.2.13: Added support for configuring cluster ip on service. This will allow you to configure a headless Kubernetes service.
• v0.2.14: Added new input parameter containerArgs that can be used to define custom args that should be passed to the deployment spec. It behaves exactly as the containerCommand, the only difference is that it will populate the args instead of the command in the spec.

Other updates

terraform-aws-load-balancer

• v0.28.3: Added support for attaching additional secondary certificates to the ALB listeners.
• v0.29.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.29.1: Fixed a build-related bug introduced in v0.29.0.

terraform-aws-messaging

• v0.9.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.9.1: Fixed a build-related bug introduced in v0.9.0.

terraform-aws-cache

• v0.18.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.18.1: Fixed a build-related bug introduced in v0.18.0.

terraform-aws-server

Updates we forgot to highlight in the previous newsletter:

• v0.14.3: Fixed default route conflict when attaching multiple ENIs on Ubuntu.
• v0.14.4: Exposed the ability to configure permissions_boundary on each of the IAM roles created by the modules in single-server and ec2-backup. Thanks to @subashvanga-mf.
• v0.14.5: Added new get_password_data parameter in single-server. Thanks to @fishhead108.
• v0.14.6: Added host_id var to aws_instance resource in single-server that will allow it to be created on a dedicated EC2 host.

Updates from the most recent period:

• v0.14.7: Fixed a bug where IAM roles that were created through the AWS console could not be passed to the single-server module because of duplicate instance profile. Allows disabling the creation of instance profile internally inside the module.
• v0.14.8: Replaced hardcoded IAM arn with a data source, for compatibility with multiple partitions, in ec2-backup. Thanks to @dkaushik100.
• v0.15.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.15.1: Fixed a build-related bug introduced by v0.15.0.

terraform-aws-ecs

• v0.34.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.34.1: Updated ecs-cluster module to use the aws_ecs_cluster_capacity_providers to avoid the need for a python script on destroy.

terraform-aws-ci

• v0.50.0: Updated the Fargate platform version to default to 1.4.0 instead of 1.3.0. Fixed bug where very short ECS tasks oftentimes end up with no streaming logs despite the logs showing up in CloudWatch. We have identified this to be a race condition between retrieving log events and the first logs to show up from the ECS task. This is addressed by adding a short delay between the ECS task starting, and the infrastructure-deployer CLI reading log events.
• v0.50.1: Updated refs for terraform-aws-ci in Dockerfiles for ecs-deploy-runner. Added patch testing on PRs within the upgrade test runner.
• v0.50.2: Added option —-no-push to docker-image-builder to perform only building of container images and avoid pushing to registry.
• v0.50.3: Bug fix in the upgrade tester code that we use internally to validate our releases are backward compatible.
• v0.50.4: Updated ecs-deploy-runner to support tagging of multiple images.
• v0.50.5: Made more improvements to the upgrade tester code that we use internally to validate that our releases are backward compatible.
• v0.50.6: Added support in build-docker-image ECS Deploy Runner script for injecting docker buildkit compatible secrets into Kaniko builds via the --env-secret parameter.
• v0.50.7: Updated terraform-update-variable to make the formatting step optional, allowing you to run it without terraform being available in the PATH.

terraform-aws-ci-steampipe

• v0.3.0: Updated various dependencies — refer to the release notes for more details.
• v0.3.1: Updated the underlying go dependency versions of the run-steampipe-mod-check trigger command for the steampipe-runner. Fixed bug where --publish-to-securityhub did not work due to security token failures when running in the ECS Deploy Runner. Added new option --report-global-to-region which allows publishing global findings (e.g., issues with IAM) to Security Hub in the specified region.
• v0.3.2: Added new option --ensure-iam-credential-report which will ensure that a completed credential report is available for Steampipe to query.
• v0.3.3: Updated default version of steampipe to latest patch release: v0.15.0 => v0.15.3
• v0.3.4: Updated dependencies — refer to the release notes for more details.

terraform-aws-eks

• v0.52.0: Added the ability to restrict IMDS on self managed workers. Additionally, the launch template is now updated to restrict the service to require tokens by default (IMDS v2).
• v0.53.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.

terraform-aws-data-storage

• v0.24.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.24.1: Added the ability to configure the character sets for the RDS Database for Oracle and MSSQL types.
• v0.24.2: Fixed bug where aurora module could not be configured with a restore_to_time setting due to mutual exclusivity.

terraform-aws-vpc

• v0.22.2: Removed the log:CreateLogGroup permissions from the VPC Flow Logs. This was unnecessary as the Log Groups are managed in Terraform, and in fact can lead to a race condition where the Log Group gets recreated after deletion.
• v0.22.3: Added a new input variable exclude_ports_from_inbound_all that can be used to exclude a list of ports from the inbound global CIDR rules in the network acl modules. This is useful for adhering to certain compliance standards like CIS that explicitly deny any allow rule for administrative ports.

terraform-aws-security

• v0.65.7: Added partial support for deploying AWS Config in previously unsupported region ap-northeast-3.

terraform-aws-monitoring

• v0.35.0: Updated sns-to-slack module to require passing through the Slack webhook URL using AWS Secrets Manager instead of directly as module variables. This is to treat the webhook URL more like a Secret as recommended by Slack.
• v0.35.1: Added the ability to configure the SNS Topic with cross account access for publish and list subscriptions.
• v0.35.2: Add variable to configure how to treat missing data for Memcached and Redis alarms
• v0.35.3: Renamed legacy vars.tf files to variables.tf.
• v0.35.4: Updated all alarms module to expose treat_missing_data as a configurable parameter.

terraform-aws-static-assets

• v0.15.7: Added support for associating CloudFront functions with the CloudFront distribution managed with the s3-cloudfront module; Updated the website S3 bucket with additional security configurations when operating in private bucket mode.

terraform-aws-lambda

• v0.20.0: Unlock AWS provider v4. Require minimum 3.75.1. This update includes a few tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide notes for more info.
• v0.20.1: Add patch that automates upgrading from v0.16.x to v0.17.0. Adds tracing_config variable. Update aws_arn data call to use data partition.
• v0.20.2: Added the ability to configure multiple secrets manager entries for container based Lambda functions in the run-lambda-entrypoint command.

terraform-aws-openvpn

• v0.24.3: Update dependencies of openvpn-admin utility to support usage with AWS SSO.

gruntkms

• v0.0.11: Updated decryption routine to decrypt multiple ciphertexts in the input concurrently to improve performance.

DevOps News

CDK for Terraform is now generally available

What happened: HashiCorp has announced that CDK for Terraform (CDKTF) is now generally available (GA).

Why it matters: CDKTF allows you to use Terraform without using the HashiCorp Configuration Language (HCL). That is, you can manage your infrastructure code using the same general purpose programming language you use for your apps (supported languages include TypeScript, Python, C#, Java, and Go) while still benefiting from the fact that CDKTF has full interoperability with HashiCorp’s ecosystem of Terraform providers and modules. Now that CDKTF is GA, it may be stable enough to start experimenting with it for production use cases.

What to do about it: Give CDKTF a shot and let us know what you think! Should Gruntwork have first-class support for CDKTF? What would you like to see here?