Gruntwork Marketing Site

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

Happy new year! The Gruntwork team was on vacation for a couple weeks, but we’re back now, and have lots of fun updates to share. In the last month, we wrote a year in review blog post where we discussed what we did in 2018 and our roadmap for 2019, added support for automated, zero-downtime AMI rollouts to our Kubernetes modules, added support to Terratest for testing Kubernetes, added modules to define CloudWatch Dashboards as code, and made lots of other updates and bug fixes. In other news, AWS now offers Amazon DocumentDB, a managed document data store with MongoDB compatibility, and Amazon VPN, a managed VPN service, plus Fargate is ~50% cheaper and Amazon ElasticSearch is now HIPAA eligible and PCI and ISO compliant.

As always, if you have any questions or need help, email us at support@gruntwork.io!

Gruntwork Updates
DevOps News
Security Updates

Gruntwork Updates

A year in review, 2018

Motivation: We wanted to take some time to pause, reflect on what we did, and think a little about the future of Gruntwork.

Solution: We wrote a blog post called A Year in Review, 2018, where we summarized what we did at Gruntwork in 2018 and what we’ll be working on in 2019.

What to do about it: Read the blog post and share your feedback, especially on our roadmap for 2019!

EKS Infrastructure Package now supports automated AMI rollouts

Motivation: When you need to update the underlying servers of an EKS cluster (e.g to upgrade the kernel version), you want to minimize service disruption to your running Pods. A naive approach of launching new instances and terminating the old ones would not take into account the Pods that are already scheduled and running on your old instances.

Solution: We created a new subcommand in kubergrunt to cover how to perform a zero downtime update to your Kubernetes cluster. The subcommand performs the grunt work of expanding the nodes in your cluster and migrating the Pods off of the old nodes to the new ones. You can read more about the subcommand in the docs.

There were also several other package-k8s updates:

v0.1.2: Verified support for Windows Powershell environments. Added the ability to specify the underlying Kubernetes version when launching your EKS cluster.
v0.1.4: When provisioning an EKS cluster, we will now wait for the API endpoint to come up before marking the resource as created. This addresses an eventual consistency issue where chaining eks cluster resources to kubernetes resources could sometimes fail because terraform marks a resource as complete before the API comes up and is responsive. Note that this requires installation of kubergrunt. You can get back to the older behavior by setting use_kubergrunt_verification to false in the module parameters.

What to do about it: Grab package-k8s, v0.1.4 and try out the new subcommand in kubergrunt. This repos is currently in private beta and is open to any Gruntwork subscriber. If you are interested in participating, email us at support@gruntwork.io and we’ll grant you access (and if you’re not a subscriber, sign up now)!

Terratest now supports automated testing of Kubernetes

Motivation: Terratest is Gruntwork’s swiss army knife for infrastructure testing. With a fast moving landscape like DevOps, it is important to have automated tests for your infrastructure code to ensure that your code still works across upgrades and API changes. Last month, we began adding functionality to support testing Kubernetes clusters. We continue to expand our Kubernetes offering in Terratest this month as well.

Solution: Terratest now supports end to end testing of a Kubernetes deployment! Check out kubernetes_basic_example_test.go and kubernetes_basic_example_service_check_test.go for examples.

Other Terratest updates:

v0.13.16

Fixes a bug in RunDummyServer where you could not spawn multiple servers concurrently.
Introduces GetRandomStableRegion , which will only select from a list of regions that have been around for more than a year.
Introduces ContinuouslyCheckUrl , which can be used to periodically ping a given URL and check for 200 status code.

v0.13.17

Fixes a bug in shell module where the stdout and stderr of the subcommand was not streaming properly.

v0.13.18

Added GetPrivateIpOfEc2Instance and GetPrivateIpsOfEc2Instances methods that return the private IPs of EC2 instances.

v0.13.19

Added support on terraform module for returning all terraform output or specific ouput keys as a map.

v0.13.20

Added GetCapacityInfoForAsg to get current capacity information about an ASG given its name.
Added WaitForCapacity for waiting for ASG to reach the desired capacity.
Added GetPrivateHostnameOfEc2Instance, and GetPrivateHostnamesOfEc2Instances methods that return the private DNS name of the EC2 instance

What to do about it: Grab Terratest v0.13.20 and take the k8s module for a spin!

Define CloudWatch Dashboards as code

Motivation: Several customers wanted a way to automatically create CloudWatch Dashboards that show all their most important metrics in one place, but instead of creating those Dashboards manually, they wanted a way to define them as code.

Solution: We’ve added several modules to module-aws-monitoring to make it easy to define CloudWatch Dashboards with Terraform. Here’s an example:

# Create a widget that displays some text
module "text_widget_1" {
source = "git::git@github.com:gruntwork-io/module-aws-monitoring.git//modules/metrics/cloudwatch-dashboard-text-widget?ref=v0.10.2"

markdown = "A text widget"
}

# Create a widget that shows some metrics about an EC2 Instance
module "metric_widget_1" {
source = "git::git@github.com:gruntwork-io/module-aws-monitoring.git//modules/metrics/cloudwatch-dashboard-metric-widget?ref=v0.10.2"

metrics = "${list(
list("AWS/EC2", "CPUUtilization", "InstanceId", "i-abc"),
list("AWS/EC2", "CPUUtilization", "InstanceId", "i-xyz"),
)}"
}

# Put all of the widgets into a dashboard
module "dashboard" {
source = "git::git@github.com:gruntwork-io/module-aws-monitoring.git//modules/metrics/cloudwatch-dashboard?ref=v0.10.2"

aws_region = "us-east-1"
name       = "EC2UtilizationDashboard"

widgets = [
"${module.text_widget_1.widget}",
"${module.metric_widget_1.widget}",
]
}

What to do about it: These new modules are available in module-aws-monitoring, v0.10.2. Check out the cloudwatch-dashboard example for a working example!

Fixes for build args issue in sample apps in the Reference Architecture

Motivation: We recently updated the sample apps in the Reference Architecture to use Docker build args, but forgot to update docker-compose.yml and the CI/CD scripts, so docker-compose up and the CI/CD jobs were both failing due to missing --build-arg arguments.

Solution: To fix this issue, you need to make a few small tweaks in your sample apps and any of your real world apps that you based on the sample apps. Use the following commits for reference:

Here are the steps:

Update your apps to Docker Compose version 3 by adding version: '3' at the top of your docker-compose.yml and putting all of the services under the services: key (example). We need the latest version of Docker Compose to support build args.
In the build section of each service, you can pass build args by adding an args section that passes through your GitHub personal access token as the environment variable GITHUB_OAUTH_TOKEN (example).
In your CI scripts, add --build-arg GITHUB_OAUTH_TOKEN="$GITHUB_OAUTH_TOKEN" to any call to build-docker-image (example).
Optional: in Docker Compose version 3, links are deprecated, so we recommend migrating off of them. To do that, replace links with depends_on and update your code to use hostnames set by Docker Compose networking in dev and your own (simpler) environment variables in prod (app example, Terraform example).
Optional: you may need to update the version of Docker in your CI server too. We used to install 17.03.1~ce-0~ubuntu-xenial but to use intermediate Docker builds and build args, you need something newer, such as 18.06.1~ce~3–0~ubuntu.

What to do about it: If we deployed a Reference Architecture for you between November 12th and December 12th, 2018, we strongly recommend following the steps above to fix these issues! If you have any questions, contact us via support@gruntwork.io or our shared Slack channel.

Other open source updates

terraform-aws-vault, v0.11.2: Removes pip and boto3 installation from the install-vault module and adds it to the packer template example with an optional flag. Bumps Vault version to 1.0.0 at the packer template for the ami example and uses Vault open source for running the auto-unseal example tests. Additionally, run-vault module was updated to support for configuring KMS API endpoint for auto-unseal, and formatting that caused unexpected behavior was removed.
terraform-aws-vault, v0.11.3: Fixes error when configuring auto unseal endpoint.
terraform-google-consul, v0.3.2: Allows run-consul to override default autopilot configuration.
terraform-google-vault, v0.1.3: install-vault now supports installing Vault and Consul from an arbitrary URL, which allows installing Vault and Consul enterprise. vault-cluster now allows using a different project id for the base compute image, and has support for generating its own service account to operate to operate the Vault cluster. The repository also now has tested examples on installing Vault Enterprise, Auto-unsealing a Vault cluster and authenticating to Vault with GCP using both the both the iam and gce methods, and run-vault now supports enabling auto-unsealing and setting the necessary KMS key for it. Lastly, a couple of bugs were fixed related to missing permissions.
Terragrunt, v0.17.4: Added support for a --terragrunt-include-dir argument that you can use to specify the folders to include for terragrunt xxx-all commands (rather than all sub-modules).

Other updates

package-lambda, v0.4.0: Fixes a perpetual diff issue where terraform always detected a change when no files in the lambda package had changed.
package-lambda, v0.5.0: The archive file generated by the lambda and lambda-edge modules no longer default to being created in the source_dir . This caused a previous run’s zip archive to be added to the zip file.
module-data-storage, v0.8.3: Upgrades underlying lambda functions for snapshot management to use package-lambda v0.5.0 to fix a perpetual terraform plan diff issue.
module-data-storage, v0.8.4: Fixes an issue with the parameter group name when trying to create a Postgres 10 database using the rds module.
module-data-storage, v0.8.5: You can now configure the aurora module to export logs to CloudWatch using the new enabled_cloudwatch_logs_exports input parameter.
module-server, v0.5.2: Fix a bug with how the mount-ebs-volume script checked if a volume was already formatted.
module-server, v0.5.3: You now set the new input parameter allow_all_outbound_traffic to false to not allow all outbound traffic from the Security Group created by the single-server module.
module-server, v0.5.4: The mount-ebs-volume script will now retry correctly if an EBS volume exists but is attached to a different EC2 instance. This is useful to ensure the script retries while an old instance shuts down and releases the volume.
module-ci, v0.13.5: The jenkins-server module now exposes all its health check settings via new input parameters.
module-ci, v0.13.6: The jenkins-server module now exposes a ebs_volume_snapshot_id param to allow you to restore from an EBS snapshot.
module-ci, v0.13.7: The git-add-commit-push script will now retry on the "failed to update ref" error, which seems to come up intermittently.
module-asg, v0.6.20: The ebs_volumes parameter in the server-group module now allows you to specify snapshot_id to force an EBS volume to restore from a snapshot rather than loading one from scratch.
module-asg, v0.6.21: Fix an issue where destroying aserver-group would cause an error.
module-asg, v0.6.22: Fix a bug where you’d get an error if you passed more than one CIDR block into the allow_ssh_from_cidr_blocks parameter.
package-mongodb, v0.3.0: install-mongodb no longer tries to upgrade pip, as this causes issues with pip disappearing from the PATH. We recommend removing pip upgrade steps from your Packer templates too.
module-security, v0.15.6: The cloudtrail module now grants key administrators the kms:Tag* and kms:Untag* permissions.
package-messaging, v0.1.2: The sqs module now has support to passing down tags that will be applied to the resources.
package-messaging, v0.1.3: The sqs module now allows you specify a KMS key for encrypting messages and to optionally disable the ip access policy.
pa ckage-zookeeper, v0.4.10: Update zookeeper-cluster to address potential issues when destroying resources.
package-kafka, v0.4.4: Big improvements in testing: Tests are upgraded to terratest v0.13.20 and tests are now executed using 3 different Linux distributions: Ubuntu, AmazonLinux and CentOS. Also, a number of module dependencies were updated to more recent versions.
module-vpc, v0.5.2: You can now create Elastic IP Addresses (EIPs) for your NAT Gateways outside of the vpc-app module and tell the module to use those EIPs by setting the use_custom_nat_eips parameter to true and passing in the list of EIP allocation IDs using the custom_nat_eips parameter.
module-vpc, v0.5.3: You can now get the NAT Gateway IDs from the vpc-app module using the new nat_gateway_idsoutput attribute.
package-terraform-utilities, v0.0.5: Introduces a new module require-executable that can be used to ensure particular executables is available in the PATH, with a customizable error message when it is not found.
package-terraform-utilities, v0.0.6: This release introduces modules (run-pex-as-data-source and run-pex-as-resource) that support running python PEX files in Terraform in a way such that the scripts themselves do not need to be embedded in the pex. See the documentation to learn more about pex.

DevOps News

Amazon DocumentDB (with MongoDB compatibility)

What happened: Amazon has announced DocumentDB, a document database that is compatible with MongoDB.

Why it matters: MongoDB is a beast to operate, with lots of complexity in how you configure it, deploy it, upgrade it, back it up, scale it, and so on. Having a managed service do this for you could be a huge win.

What to do about it: Check out the product page for more info.

AWS Managed site-to-site and client VPN services

What happened: AWS has launched an AWS VPN Service, which provides managed site-to-site VPN, for connecting your office and data centers to AWS, and client VPN, for allowing users to connect to your private networks.

Why it matters: Before, you’d have to run all the VPN appliances yourself. For example, package-openvpn allowed you to run an OpenVPN server yourself, which allowed a lot of customization, but also meant you were on the hook for configure the server, deploying it, keeping it running, updating it, backing it up, and so on.

What to do about it: Check out the AWS VPN product page for more info.

AWS Fargate ~50% Price Reduction

What happened: AWS has announced that they are reducing the price for AWS Fargate by 20% for vCPU and 65% for memory.

Why it matters: Fargate is exciting in that it allows you to run Docker containers on AWS without having to manage the underlying servers, but one of the biggest drawbacks was its cost, which could be 2x (or more!) the price of running Docker containers directly on EC2 Instances that you manage. Now that the price is coming down by roughly ~50%, this makes Fargate an even more interesting option.

What to do about it: The price reduction is already in effect!

Amazon Elasticsearch Service is now HIPAA eligible and PCI and ISO compliant

What happened: Amazon has announced that its managed Elasticsearch Service is now HIPAA eligible and in-scope of AWS’ PCI DSS and ISO 9001, 27001, 27017, and 27018 certifications.

Why it matters: This allows you to use Amazon Elasticsearch Service for a large variety of use cases that require these certifications, including healthcare (i.e., to store and analyze protected health information or PHI) and payment processing (i.e., to store, process, or transmit credit card data).

What to do about it: Check out the Amazon Elasticsearch product page for more info.

Security Updates

Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.

Kubernetes

CVE-2018–18264: The Kubernetes Dashboard before version 1.10.1, by default, has elevated privileges, and allows attackers to bypass authentication, giving the attacker read-only access to the cluster, including all secrets in the cluster. This vulnerability has been around for more than a year and exploited in the wild (e.g., Tesla’s infrastructure was hijacked for cryptocurrency mining), but as we ramp up our Kubernetes usage, we thought it best to notify our customers that (a) you should update your Kubernetes Dashboard version and (b) you should NOT expose your Kubernetes Dashboard to the public Internet.

Enjoyed the article?

Gruntwork Newsletter, January 2019