Gruntwork Newsletter, December 2020

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

In the last month, we finished updating all our modules to work with Terraform 0.13, began the upgrade process for Terraform 0.14, got most of the CIS v1.3 upgrade done, added a new module for managing AWS Secrets Manager policies, published a style guide for Go, made our VPC module more flexible, and made a huge number of other fixes and improvements. In the wider DevOps world, AWS had its re:Invent conference, and announced dozens of new services and features, including managed Prometheus, managed Grafana, Aurora Serverless v2, and much more. Finally, a reminder that we’ll be taking a holiday break for two weeks in December as usual, so please plan accordingly!

As always, if you have any questions or need help, email us at support@gruntwork.io!

• Gruntwork Updates
• DevOps News
• Security Updates

Gruntwork Updates

It’s time to update to Terraform 0.13!

Motivation: A few months ago, HashiCorp released version 0.13.0 of Terraform. It brought lots of new features, but also included several backwards incompatible changes that made upgrading difficult. Last month, we announced that most of our modules had been upgraded, but a few items still remained.

Solution: We’ve finished the Terraform 0.13 upgrade and it’s ready for you to take it for a spin! We’ve tested all our modules with Terraform 0.13, fixed all the incompatibility issues we ran into, released new versions of all the repos, updated the Acme Reference Architectures, and put together a migration guide for you to follow (which includes a version compatibility table and links to the Acme Reference Architecture upgrades to follow as examples).

What to do about it: You can now update all of your repos to use Terraform 0.13 and the new versions of the Gruntwork repos. Follow our migration guide and let us now how it goes! Please note that Gruntwork will be on holidays starting next week (see the entry below), so if you hit issues with the upgrade, we’ll only be able to help you when we’re back in early January!

Terraform 0.14 update

Motivation: A couple weeks ago, HashiCorp released version 0.14.0 of Terraform. It is possible that Terraform is speeding up its release cycle, as there was nearly 2 years between 0.11 and 0.12, a little over 1 year between 0.12 and 0.13, but less than 4 months between 0.13 and 0.14.

Solution: We have started the process to test all our repos with Terraform 0.14. Unlike the last two releases, 0.14 seems to be mostly backwards compatible, so we are hoping this upgrade will go much faster, and that we’ll have something available in January.

What to do about it: For now, sit tight. We’ll notify everyone when the Terraform 0.14 upgrade is ready!

CIS AWS Foundations Benchmark v1.3

Motivation: A few months ago, the Center for Internet Security (CIS) released version 1.3.0 of the AWS Foundations Benchmark. This introduced several new controls that we needed to incorporate into our Gruntwork Compliance for CIS product. We started working on this upgrade last month.

Solution: We are making good progress on the upgrade and are nearly finished:

There items remaining are are two coding tasks (IAM access analyzer, and NACL changes), some documentation updates, and writing up a migration guide. We hope to have this ready for use in January.

What to do about it: For now, sit tight. We’ll notify everyone when the CIS v1.3 upgrade is ready!

New module: secrets-manager-resource-policies

Motivation: Managing sensitive data in Terraform is a tricky business. To see just how tricky, refer to our blog post entitled A comprehensive guide to managing secrets in your Terraform code. One of the challenges of managing sensitive data is how to manage the access to the secret in addition to the value of the secret.

Solution: Secrets Manager is a service from AWS designed specifically to manage secrets. Terraform has several resources and data sources for working with Secrets Manager secrets, including the [aws_secretsmanager_secret_policy resource](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/secretsmanager_secret_policy) which uses an IAM policy to allow access to the secret. The new [secrets-manager-resource-policies module](https://github.com/gruntwork-io/module-security/blob/v0.41.3/modules/secrets-manager-resource-policies/README.md) has default policies for read only and full access to secrets. Note that this module is not used for managing secret values, only secret policies. Managing secrets values with Terraform is not recommended because the value will exist in plain text in the Terraform state file.

What to do about it: Check out the new module and the corresponding example and start managing your secret policies with code.

New guide: Go style guide

Motivation: When writing code in any programming language, it’s helpful to have a set of guidelines and conventions to adhere to, in order to follow a set of best practices and ensure that the code is clear, readable and idiomatic. Many programming languages already have such styles guides that are commonly known and used and are de facto industry standard — in the case of Go, that is Effective Go. At Gruntwork, we also have some idiosyncrasies that are particular to our code and repos.

Solution: We wrote a style guide for Go that uses Effective Go as a starting point and extends it to add guidelines that are particular to Gruntwork.

What to do about it: Check out the new Go style guide in the Guides section on our website!

A more flexible vpc-app module!

Motivation: We had a number of requests from customers to make the vpc-app module more flexible in terms of the subnet tiers (public, private-app, private-persistence) it creates: for example, some customers wanted to be able to disable the public subnet tier entirely, as they already had public Internet access through another mechanism (e.g., Direct Connect, VPC peering, etc) and didn’t need the extra subnets, NAT gateways, Internet gateways, etc.

Solution: We’ve updated the vpc-app module so that you can now disable any of the three tiers of subnets (public, private-app, private-persistence) by setting the new input variables create_public_subnets, create_private_app_subnets, or create_private_persistence_subnets to false. For example, here’s how you can create a VPC with the public subnets disabled, so it only has the private-app and private-persistence subnets:

module "example_vpc" {
source = "git::git@github.com:gruntwork-io/module-vpc.git//modules/vpc-app?ref=v0.12.1"

aws_region       = "eu-west-1"
vpc_name         = "example-vpc"
cidr_block       = "10.0.0.0/16"
num_nat_gateways = 0

# Disable the public subnet tier
create_public_subnets = false
}

Check out the vpc-app-subnets-disabled example for the full details.

Note: As of this release, the vpc-mgmt module is now deprecated: The main difference between vpc-mgmt and vpc-app was that vpc-app had three tiers of subnets (public, private-app, private-persistence) and vpc-mgmt had two (public, private). As of this release, since vpc-app allows you to disable any of the subnet tiers, it can now support 1, 2, or 3 tiers of subnets, as needed. Therefore, we recommend using vpc-app for all your VPCs in the future. If you're already using vpc-mgmt, we will continue to maintain it for a little while longer, but please be aware that, in a future release, once we feel the new functionality in vpc-app is fully baked, we will remove vpc-mgmt entirely.

What to do about it: The new functionality is available in v0.12.1 of terraform-aws-vpc. Give it a try and let us know what you think!

Winter break, 2020

Motivation: At Gruntwork, we are a human-friendly company, and we believe employees should be able to take time off to spend time with their friends and families, away from work.

Solution: The entire Gruntwork team will be on vacation December 21st — January 1st. During this time, there may not be anyone around to respond to support inquiries, so please plan accordingly. We’ll be back on January 4th and ready to take on the new year! Note: we previously announced this in last month’s newsletter and via Community Slack.

What to do about it: We hope you’re able to relax and enjoy some time off as well. Happy holidays!

Open Source Updates

Terratest

• v0.30.23: Add another flavor of plugin retrieval error to list of default retryable errors.
• v0.30.24: New functions helm.AddRepo and helm.RemoveRepo for adding and removing helm repositories from the CLI configuration. This also includes a new function aws.DeleteParameter for deleting SSM parameters.
• v0.30.25: Packer functions now support passing through ‘-except’ arg.
• v0.30.27: Fix a bug in the availability checking logic of k8s.IsPodAvailable where it would incorrectly report pods as available when the readinessProbe is failing but the containers were in a started state.
• v0.31.0: Update Terratest to work with Terraform 0.14. In particular, fixed a bug with the terraform.Output function where it was not correctly parsing the format returned by the terraform output command in 0.14.
• v0.31.1: Add sovereign Cloud Support for the azure modules. Update the ssh module to use net.JoinHostPort to build addresses so it works with IPv6 addresses too. Fix the terraform.Output method so it works with all output types and not just strings.
• v0.31.2: You can now pass through additional arguments to helm commands such as Install and Upgrade by using the new ExtraArgs attribute on the helm.Options struct. Add support for Azure Recovery Services. Refer to the recoveryservices.go file in the azure module for information on supported functions.

Terragrunt

• v0.26.3: Fix a bug where Terragrunt would pass through the TF_CLI_ARGS environment variable when calling terraform --version for an internal version check, which could lead to errors. Terragrunt will now omit this environment variable during this version check.
• v0.26.4: The xxx-all commands (e.g., apply-all) now skip the .terragrunt-cache entirely when scanning for Terragrunt modules, which should make these commands run faster.
• v0.26.5: The xxx-all commands (e.g., apply-all) now properly skip the .terragrunt-cache on Windows.
• v0.26.6: The gcs configuration for remote_state now supports impersonate_service_account when creating the bucket.
• v0.26.7: Fix bug where mock_outputs_allowed_terraform_commands did not work correctly for nested dependencies.

cloud-nuke

• v0.1.24: Now it’s possible to nuke Lambda resources.

kubergrunt

• v0.6.5: New commands for cleaning up security groups and scheduling CoreDNS on Fargate and EC2 worker types.
• v0.6.6: Minor change to increase the timeout on cleaning up resources in AWS.
• v0.6.7: Bug fix that improves error handling for an edge case discovered in testing, introduced in v0.6.5.
• v0.6.8: Bug fix for handling case where Network Interface is already detached in eks cleanup routine.

pre-commit

• v0.1.12: The terraform-validate hook will now (a) set the TF_IN_AUTOMATION variable to reduce Terraform output that isn't relevant in automation, (b) print out each directory it's running in so if you hit an error, you know where to look, (c) save errors until the end, so validate runs in all modules, rather than exiting on the first error. The terraform-fmt hook will now (a) run with -diff -check so the differences and affected files are printed to logs, rather than written to disk and (b) save errors until the end, so fmt runs in all modules, rather than exiting on the first error.

terraform-google-sql

• v0.3.1: The cloud-sql module now supports the point_in_time_recovery_enabled option for Postgres databases.

Other updates

module-security

• v0.41.2: You can now configure how many days to retain CloudWatch logs in the cloudtrail module using the new num_days_to_retain_cloudwatch_logs input variable.
• v0.41.3: Adds a new module, secrets-manager-resource-policies , described earlier in this blog post.
• v0.43.0: The aws-config-bucket module has been refactored to use the private-s3-bucket module under the hood to configure the S3 bucket.
• v0.44.0: The cloudtrail-bucket module has been refactored to use the private-s3-bucket module under the hood to configure the access logging S3 bucket.
• v0.44.1: You can now configure the bucket ownership settings using the new bucket_ownership input variable in private-s3-bucket.
• v0.44.2: kms-grant-multi-region now supports using aliases for KMS Key IDs.
• v0.44.3: This fixes a perpetual diff issue with cloudtrail module when kms_key_arn is a loose KMS ID (e.g., KMS Alias).
• v0.44.4: This fixes a bug that was introduced in v0.44.3, where the cloudtrail module now needed kms:DescribeKey access to the KMS key, which was not provided by default. This release reverts back to the behavior in v0.44.2 , and introduces two new feature flags to control the behavior with aliases: allow_kms_describe_key_to_external_aws_accounts and kms_key_arn_is_alias . This release also allows you to attach kms:DescribeKey permissions to IAM entities on CMKs managed with kms-master-key by setting cmk_describe_only_user_iam_arns.
• v0.44.5: This release adds support for configuring data event logging for cloudtrail buckets. Data event logging is configured using the newly introduced variables: data_logging_enabled, data_logging_read_write_type, data_logging_include_management_events, data_logging_resource_type and data_logging_resource_values. For detailed instructions see the descriptions of these variables.

terraform-aws-monitoring

• v0.23.4: Only create the RDS high replica lag alarm in the rds-alarms module if there is at least one replica (num_rds_instance_ids is greater than 0).
• v0.24.0: The load-balancer-access-logs module has been refactored to use the private-s3-bucket module under the hood to configure the access logging S3 bucket.

package-messaging

• v0.4.1: You can now specify custom tags to apply to the Kinesis stream using the new tags input variable.

package-static-assets

• v0.7.1: You can now get the ARN of the CloudFront distribution using the new cloudfront_distribution_arn output variable.

module-ci

• v0.29.2: The default version of tools used in the Docker image for the ECS Deploy Runner has been updated to the latest versions: terraform is now 0.12.29 , terragrunt is now 0.26.4 , packer is now 1.6.5 , and kubergrunt is now 0.6.4 .
• v0.29.3: The default version of tools installed in the ecs-deploy-runner docker containers have been updated: module_ci_version is now v0.29.2, and kaniko is now v1.3.0.
• v0.29.4: This release fixes a bug in build-packer-artifact script, where the --idempotency flag did not properly handle images with multiple tags.

terraform-aws-eks

• v0.28.0: This release updates the eks-alb-ingress-controller to use the new chart location following the deprecation of the incubator and stable helm chart repository. In the process, the underlying controller has been upgraded to v2. Please refer to the migration guide for information on updating to this release.
• v0.29.0: The cleanup routine for EKS control plane will now cull Security Groups created by the AWS Load Balancer Controller. This release also updates eks-cluster-workers to allow you to specify different instance types for each ASG specified in var.autoscaling_group_configurations. As part of this change, var.autoscaling_group_configurations was converted from an object type with concrete attributes to an any to allow for optionality in the attributes. Now you only need to specify subnet_ids as opposed to the whole object, with the missing values being sourced from the variables prefixed with asg_default. Refer to the updated variable documentation for more details. This is a backwards incompatible release - refer to the migration guide for information on updating to this release.
• v0.29.1: The type of pod_tolerations input var was incorrect for eks-alb-ingress-controller, eks-k8s-cluster-autoscaler, eks-k8s-external-dns. eks-cluster-managed-workers now supports specifying launch templates.
• v0.29.2: You can now set the capacity_type on the Managed Node Groups created with eks-cluster-managed-workers.
• v0.30.0: terraform-aws-eks is now compatible with Terraform 0.13.x! Some variables and virtual resources have been removed. The release notes will guide you through the migration.
• v0.31.0: Various instance parameters are now overrideable in the autoscaling_group_configurations. Refer to the updated variable definition for more details on which attributes are available to override.
• v0.31.1: This release is a minor bugfix to use the latest kubergrunt (v0.6.8) required dependency.

module-data-storage

• v0.17.0: The efs module now allows you to grant root access to the EFS volume using the root_access_arns field in the efs_access_points input variable.
• v0.17.1: You can now tell the rds and aurora modules to ignore changes to the master_password parameter by setting the new ignore_password_changes input variable to true. This is useful when managing the password outside of Terraform, such as with auto-rotating passwords in AWS Secrets Manager.

module-ecs

• v0.23.1: You can now configure the permissions boundary for the auto scaling IAM role for in ecs-service using the new autoscaling_role_permissions_boundary_arn input variable.
• v0.23.2: You can now configure the ecs-cluster to create one capacity provider and one ASG per AZ / subnet by setting the multi_az_capacity_provider input variable to true.
• v0.23.3: You can now enable container insights on the ECS cluster deployed with the ecs-cluster module.

package-lambda

• v0.9.4: You can now configure a custom assume role policy for the IAM role in the lambda module using the new assume_role_policy input variable.

package-zookeper

• v0.8.0: The exhibitor-shared-config module has been refactored to use the private-s3-bucket module under the hood to configure the S3 bucket.

package-openvpn

• v0.12.1: With this release package-openvpn now supports Ubuntu 20.04 LTS!
• v0.13.0: The openvpn-server module has been refactored to use the private-s3-bucket module under the hood to configure the S3 bucket.

terraform-aws-vpc

• v0.12.0: The vpc-flow-logs module has been refactored to use the private-s3-bucket module under the hood to configure the S3 bucket.
• v0.12.1: The vpc-app module now allows you to disable any of the three tiers of subnets (public, private-app, private-persistence) by setting the new input variables create_public_subnets, create_private_app_subnets, or create_private_persistence_subnets to false. This is convenient, for example, if you want to create a VPC with no public subnets because you get all public Internet access through some other mechanism (e.g., Direct Connect, VPC peering, etc).
• v0.12.2: Fix a bug in how the vpc-flow-logs module looked up the KMS key when create_resources was set to false.
• v0.12.3: The vpc-app module now allows you to configure the ingress and egress rules for the default Security Group and NACL using the new default_security_group_ingress_rules, default_security_group_egress_rules, default_nacl_ingress_rules, and default_nacl_egress_rules input variables. You can also control tags on these resources using the existing custom_tags input variable.
• v0.12.4: Fix a bug where vpc-app-network-acls would not work correctly if some of the subnet tiers in the VPC were disabled.

module-server

• v0.9.4: Replace template_file usage with locals to avoid data source dependency graphs.

DevOps News

AWS updates (including re:Invent)

AWS has announced a huge number of new releases in the last month, partially as part of their annual re:Invent conference. Below is a list of just a few of the highlights. For the full details, check out AWS News.

• Managed Prometheus Service: Prometheus is a popular open source monitoring and alerting solution, and AWS will now have a managed version of it so you don’t have to run it yourself!
• Managed Grafana Service: Grafana is a popular open source data visualization and querying tool that is often combined with Prometheus, and now AWS has a managed version of both of these services!
• Managed EKS add-ons: You can now manage the version of EKS components (e.g., kube-proxy, coredns, and aws-vpc-cni) through the EKS API. This may reduce the need for using external tooling to manage these items.
• Built-in logging for EKS Fargate: EKS Fargate Tasks can now automatically ship logs to Amazon Cloudwatch, Amazon Elasticsearch, Amazon Kinesis Data Firehose, and Amazon Kinesis Streams.
• ECR replication: Amazon ECR can now automatically replicate your Docker images across AWS regions and AWS accounts.
• ECS deployment circuit breaker: Finally, Amazon ECS can automatically roll back unhealthy service deployments without the need for manual intervention.
• Lambda billing at 1ms intervals: Amazon has reduced the billing granularity for Lambda functions from 100ms down to 1ms. That means short-lived Lambda functions will cost less to operate. It also means you can rent 1ms of compute time, or as Ben Whaley put it, “I think we can say that proper utility computing has arrived.”
• Strong read-after-write consistency for S3: S3 now provides strong read-after-write consistency, which means that after a successful write of a new object, any subsequent read request immediately receives the latest version of the object.
• Amazon Aurora Serverless v2: Aurora serverless is a relational database that can automatically shut down when not in use (so you’re not billed for it), start back up when necessary, and scale capacity up or down based on your application’s needs. Amazon Aurora Serverless v2 adds the ability to scale to hundreds of thousands of transactions in a fraction of a second, plus the full breadth of Amazon Aurora’s capabilities, including Multi-AZ support, Global Database, and read replicas.
• Package Lambda functions as container images: You can now package and deploy AWS Lambda functions as a container images, rather than as Zip files.
• AWS Network Firewall: A new managed service that lets you define firewall rules for VPCs or import rules you’ve already written in common open source rule formats. It’s not yet clear how this new service compares or interacts with security groups, NACLs, route tables, Transit Gateway, etc.
• AWS Fault Injection Simulator: A managed chaos engineering service for stressing an application in testing or production environments by creating disruptive events, such as server outages or API throttling, observing how the system responds, and implementing improvements.
• AWS CloudShell: A browser-based shell available in the AWS Management Console. You can launch it with a click, it runs on top of Amazon Linux 2, has lots of handy utilities installed (including the aws CLI), and it is automatically authenticated as the IAM user you’re logged in as.
• Cross-account backups: AWS Backup now supports backing your data up to multiple AWS accounts.
• RabbitMQ support in Amazon MQ: A managed RabbitMQ service.

CentOS Linux is being sunsetted in favor of CentOS Stream

What happened: Red Hat has decided to shift focus from CentOS Linux to the new CentOS Stream.

Why it matters: If you are a CentOS user, you’ll want to decide if you’re oging to switch to CentOS Stream or something else.

What to do about it: For more information, read the blog post: https://blog.centos.org/2020/12/future-is-centos-stream/.

Security Updates

Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.

Kubernetes Man in the Middle (MITM) Vulnerability

• A new Kubernetes vulnerability was reported on December 7th that describes a MITM attack: https://discuss.kubernetes.io/t/security-advisory-cve-2020-8554-man-in-the-middle-using-loadbalancer-or-externalips/14003. We wanted to share our assessment of it here since a few media channels picked up on it (and thus you’re likely to see it in one of your feeds). The main issue of the vulnerability is the ability to launch a Service and Pod to intercept all network traffic in a cluster using the ExternalIP feature of Kubernetes. The mitigation is to disable ExternalIP using an admission hook (a Kubernetes feature to register server-side hooks that modify the resource prior to storage in etcd) that strips out this configuration or rejects it. While this is a serious vulnerability, it is mostly a problem for multi-tenant Kubernetes clusters, as this can be used to escape careful sandboxing from Namespaces and Network Security Policies. With this said, we recommend against running multi-tenant Kubernetes clusters given the amount of operational overhead required to ensure you don’t accidentally spill resources across the sandboxes. There is a mitigation hook for this vulnerability available here: https://github.com/kubernetes-sigs/externalip-webhook