Terraformer: Converting Infrastructure Into Reusable Terraform Code

13 min readAug 24, 2022

Table of Contents ·What is Terraformer?
· Why Terraformer?
· How to Use Terraformer?
 ∘ Step 1: Installation
 ∘ Step 2: Terraformer import
 ∘ Step 3: Templatize the Generated Code for Reusability
 ∘ Step 4: Standardize Terraform Modular Structure
 ∘ Step 5: Add a GitHub Actions Workflow
· Summary

The new concept of “full stack” is no longer just UI plus backend. Cloud native application development requires Infrastructure as Code (IaC) to be added to your “full stack” list. Terraformer can greatly help expand your full stack to include IaC.

What is Terraformer?

Many of us already know Terraform, the mainstream Infrastructure as Code (IaC) language. Terraformer is like the lesser-known twin of Terraform. It does exactly the opposite of Terraform — reverse engineering infrastructure into code. Simply put, Terraformer is a CLI tool that generates tf/ json and tfstate files based on your existing infrastructure.

With 8.3k stars in its GitHub repo, 130+ contributors, Terraformer, since its first public release in May 2019, has earned great popularity, leading the effort to convert existing infrastructure into Terraform code.

Why Terraformer?

Let’s face it, many of us don’t have the luxury of landing a job with state-of-the-art IaC. We find ourselves often struggling with our inherited IaC mess, with some decent Terraform code in place (if you are lucky). But often, many developers come and go with infrastructure manually built over the years. To convert your existing infrastructure into Terraform code and start your journey toward that state-of-the-art IaC target, you need to look to Terraformer.

Also, if you want to develop a Terraform template for your production resources so you can easily replicate it to multi-regions and for disaster recovery, Terraformer is the right tool for you.

Another reason why Terraformer can be of vital help is when your architect or tech lead completes a POC in your cloud provider and manually spins up a stack of services. And even better, you are handed this masterpiece when you finally get them to work seamlessly together.

At that point, your job starts, how do you replicate all that great work into another environment/account? Well, befriend Terraformer! You can reap all that stack with a few commands into beautiful Terraform code! With a few tweaks, you can templatize that generated code and reuse it over and over in different environments/accounts.

Let’s dive right in to see how this can be done.

How to Use Terraformer?

Step 1: Installation

Terraformer’s GitHub page has detailed instructions for installing Terraformer. We will follow the installation instruction from the releases.

For Windows:

Assume you already have Terraform installed: https://www.terraform.io/downloads
Navigate to Terraformer releases page, https://github.com/GoogleCloudPlatform/terraformer/releases, locate the exe file for the cloud provider and operating system you want, for example, terraformer-aws-windows-amd64.exe is for AWS as the provider and for the Windows platform.
Add the exe file path to path variable, and rename the exe file to terraformer.exe.
Run terraformer version to check whether you have it installed correctly. The latest release version, as of this writing, is v0.8.21.

For Mac:

export PROVIDER={all,google,aws,kubernetes}
curl -LO https://github.com/GoogleCloudPlatform/terraformer/releases/download/$(curl -s https://api.github.com/repos/GoogleCloudPlatform/terraformer/releases/latest | grep tag_name | cut -d '"' -f 4)/terraformer-${PROVIDER}-darwin-amd64
chmod +x terraformer-${PROVIDER}-darwin-amd64
sudo mv terraformer-${PROVIDER}-darwin-amd64 /usr/local/bin/terraformer

Step 2: Terraformer import

Let’s start with a simple main.tf file under a directory called terraformer-demo. For this demo project, we will be using AWS as our provider. main.tf has only the following simple provider configuration:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.25.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

That’s the only file we need to start using Terraformer.

Terraformer uses terraform providers, which already contain all resources and logic for working with the cloud provider’s API. A tool works with these providers through the RPC calls, so it allows them to use new versions of providers without code changes.

Now let’s run terraform init to install the plugins required for our AWS provider. If all goes well, you should be seeing output like this on your console, which indicate Terraform has been successfully initialized.

For demo purpose, I am using my AWS personal trial account, and I have configured the access key and secret key in my .aws\config file, see screenshot below, keys are masked here, replace them with your own keys.

All prep work has been completed up till now. Now let’s get to the interesting part by trying out terraformer import to do some reverse engineering of Terraform!

Import one resource

Let’s start simple. Manually create a EC2 instance in AWS console, once done, note down its instance id, in my demo, instance id is i-0e5b351eb8d11dd68, now let's execute this command:

terraformer import aws --resources=ec2_instance --filter="Name=id;Value=i-0e5b351eb8d11dd68"

In the above command, aws is the provider. --resources passes in the type of resource we want to generate Terraform code from. --filter is used to filter the resources by specific criteria, in our case, we are filtering the EC2 instance by its id i-0e5b351eb8d11dd68.

Upon running that command, we see the console output as follows:

Now let’s examine what happened in the project structure. Notice a new folder named generated got created, see highlighted portion below in the screenshot. Under generated, there is aws folder for our provider, under which is the ec2_instance folder which contains all the generated terraform code.

Open instance.tf, we see what magic Terraformer just performed!

resource "aws_instance" "tfer--i-0e5b351eb8d11dd68_demo-ec2" {
  ami                         = "ami-090fa75af13c156b4"
  associate_public_ip_address = "true"
  availability_zone           = "us-east-1a"

  capacity_reservation_specification {
    capacity_reservation_preference = "open"
  }

  cpu_core_count       = "1"
  cpu_threads_per_core = "1"

  credit_specification {
    cpu_credits = "standard"
  }

  disable_api_stop        = "false"
  disable_api_termination = "false"
  ebs_optimized           = "false"

  enclave_options {
    enabled = "false"
  }

  get_password_data                    = "false"
  hibernation                          = "false"
  instance_initiated_shutdown_behavior = "stop"
  instance_type                        = "t2.micro"
  ipv6_address_count                   = "0"
  key_name                             = "demo"

  maintenance_options {
    auto_recovery = "default"
  }

  metadata_options {
    http_endpoint               = "enabled"
    http_put_response_hop_limit = "1"
    http_tokens                 = "optional"
    instance_metadata_tags      = "disabled"
  }

  monitoring = "false"

  private_dns_name_options {
    enable_resource_name_dns_a_record    = "false"
    enable_resource_name_dns_aaaa_record = "false"
    hostname_type                        = "ip-name"
  }

  private_ip = "#.#.#.#" //masking for demo purpose

  root_block_device {
    delete_on_termination = "true"
    encrypted             = "false"
    volume_size           = "8"
    volume_type           = "gp2"
  }

  security_groups   = ["launch-wizard-2"]
  source_dest_check = "true"
  subnet_id         = "subnet-#######"

  tags = {
    Name = "demo-ec2"
  }

  tags_all = {
    Name = "demo-ec2"
  }

  tenancy                = "default"
  vpc_security_group_ids = ["sg-###########"]
}

How do we know if this generated code is indeed correct? Well, let’s find out. Let’s navigate to that directory ec2_instance, then run terraform init to initialize Terraform in this directory. Then run terraform plan, see what happened?

No changes! It shows the Terraformer generated code with the desired state matches the actual state of the infrastructure. Nice work Terraformer!

By default, the Terraformer generated code is for Terraform 0.12. We will run into the following error when we run terraform init in the generated ec2_instance folder:

Be sure to run this command to update provider in the state file. Once completed, run terraform init again, then run terraform plan.

terraform state replace-provider -auto-approve "registry.terraform.io/-/aws" "hashicorp/aws"

Import one type of resources

If we want to import one particular type of resources, such as all our Lambda functions, we can run the following command:

terraformer import aws --resources=lambda

This command will generate Terraform code for all our Lambda functions for the particular account and region we have configured.

Import more than one type of resources

If we want to import more than one type of resources, simply separate the resources with comma, such as the following command, which imports for all our Lambda functions as well as DynamoDB tables:

terraformer import aws --resources=lambda,dynamodb

Import multiple resources within the same resource type

If we need to import multiple resources, we can do so by using terraformer import as well. For example, if we need to import two Lambda functions, lambda-test1 and lambda-test2, let's try this command:

terraformer import aws --resources=lambda --filter="Name=function_name;Value=lambda-test1:lambda-test2"

Notice the filter Name is function_name, and Value has two function names separated by colon, :. Be sure to use the right syntax here. Once imported, the lambda_function.tf under generated/aws/lambda looks like the following:

resource "aws_lambda_function" "tfer--lambda-test1" {
  architectures = ["x86_64"]

  ephemeral_storage {
    size = "512"
  }

  function_name                  = "lambda-test1"
  handler                        = "index.handler"
  memory_size                    = "128"
  package_type                   = "Zip"
  reserved_concurrent_executions = "-1"
  role                           = "arn:aws:iam::##########:role/service-role/lambda-test1-role-##########"
  runtime                        = "nodejs16.x"
  source_code_hash               = "##############################="
  timeout                        = "3"

  tracing_config {
    mode = "PassThrough"
  }
}

resource "aws_lambda_function" "tfer--lambda-test2" {
  architectures = ["x86_64"]

  ephemeral_storage {
    size = "512"
  }

  function_name                  = "lambda-test2"
  handler                        = "index.handler"
  memory_size                    = "128"
  package_type                   = "Zip"
  reserved_concurrent_executions = "-1"
  role                           = "arn:aws:iam::##########:role/service-role/lambda-test2-role-##########"
  runtime                        = "nodejs14.x"
  source_code_hash               = "##############################="

  tags = {
    name = "lambda-test1"
  }

  tags_all = {
    name = "lambda-test2"
  }

  timeout = "3"

  tracing_config {
    mode = "PassThrough"
  }
}

Import resources using filters by tag

We briefly touched upon filters in the “Import one resource” section above. Let’s dive deeper here to explore how best to use filters to import code for the resources we want. We can filter resources by their tags, for example, the following command filters DynamoDB resource by its tag name with value test-table.

Notice the filter name, tags.name, this is case sensitive, if you enter upper case tags.Name, the filtered number of service may be 0 if you don't have another DynamoDB table with that tag Name defined.

terraformer import aws --resources=dynamodb --filter="Name=tags.name;Value=test-table"

Import resource with specific tag key

To filter resources with a specific tag key, say we want to filter DynamoDB with tags having tag key name defined, we can run the command like this, notice we don't need to specify Value:

terraformer import aws --resources=dynamodb --filter="Name=tags.name"

Import all resources

What if you would like to generate Terraform code for your whole infrastructure stack? The command is actually really simple:

terraformer import aws --resources=*

Yes, use wildcard to pass in to resources will do the trick. However, depending on your infrastructure size, this may take some time, and the amount of directories created under generated\aws may be astounding.

Import resource with excludes keyword

If you need to exclude certain resource from being imported, you can use --excludes keyword, sample command:

terraformer import aws --resources=* --excludes="dynamodb"

Step 3: Templatize the Generated Code for Reusability

The raw Terraform code generated by Terraformer from your existing infrastructure looks great, but it’s not that reusable as it has hard-coded values for the particular resource(s) you imported into code. In order to make the code reusable, we need to tweak it a bit. For example, in the above generated code for Lambda function, it has hard-coded Lambda function name, role, handler, etc.

If we want to reuse this code, we will need to parameterize those variables to templatize that particular resource. Here is what that Lambda function Terraform code looks like after it’s templatized. Notice those parameters starts with var, referencing their variables. For this example, we only picked a few parameters to externalize, you are welcome to parameterize the optional fields as well to allow maximum flexibility in configuring your Lambda functions.

resource "aws_lambda_function" "function" {
  architectures = ["x86_64"]

  ephemeral_storage {
    size = "512"
  }

  function_name                  = var.lambda_function_name
  handler                        = var.lambda_handler
  memory_size                    = "128"
  package_type                   = "Zip"
  reserved_concurrent_executions = "-1"
  role                           = var.lambda_role
  runtime                        = var.lambda_runtime

  tags = {
    name = var.lambda_tag
  }

  tags_all = {
    name = var.lambda_tag
  }

  timeout = "3"

  tracing_config {
    mode = "PassThrough"
  }
}

The corresponding variables.tf should look like this:

variable "lambda_function_name" {
  type = "string"
  description = "The name of the lambda function"
}

variable "lambda_handler" {
  type = string
  description = "The handler of the lambda"
  default = "index.js"
}

variable "lambda_role" {
  type = string
  description = "IAM Role"
}

variable "lambda_runtime" {
  type = string
  description = "The runtime of the lambda to create"
  default = "nodejs16.x"
}

variable "lambda_tag" {
  type = string
  description = "The tag for the lambda"
}

After you templatize the generated code, a natural question rises: how best to organize our generated Terraform code? Let’s move onto our next step.

Step 4: Standardize Terraform Modular Structure

A Terraform module is simply a collection of .tf configuration files that define multiple related resources, coded in such a way that the code can be reused. The benefit of using modules is that coding effort will be greatly reduced when doing the same thing across multiple projects.

Best practices for terraform modular structure

main.tf, variables.tf, outputs.tf. These are the recommended filenames for a minimal module, even if they're empty. main.tf should be the primary entrypoint.
The root directory of the project can be referred to as the “root module”. Nested modules, aka “child modules”, should exist under the modules/ subdirectory. The code in the root module usually calls the child modules as required.
These modules can be nested, although it is recommended not to go more than 3/4 levels deep and to avoid this if possible to reduce complexity.
The root module and any nested modules should have README files.

A sample modular Terraform project structure looks like this:

Notice the multiple modules under modules directory. A recommended approach is to move this modules directory to a shared repository so all projects within your organization can call those centralized templatized reusable modules from their individual projects.

Also notice the .tfvars files located under the environment related directories such as dev and prod under .env. These are the files holding environment specific values for the variables.

To continue our discussion above on templatized Lambda module, we now can have the root main.tf call the Lambda reusable module to create two new Lambda functions lambda_demo1 and lambda_demo2, see snippet from our root main.tf below:

module "lambda_function_demo1" {
  source        = "./modules/lambda"
  function_name = "lambda_demo1"
  role          = var.lambda_iam_role_arn
  handler       = "index.js"
  filename      = "../lambda-demo1/deploy.zip"
}
module "lambda_function_demo2" {
  source        = "./modules/lambda"
  function_name = "lambda_demo2"
  role          = var.lambda_iam_role_arn
  handler       = "index.js"
  filename      = "../lambda-demo2/deploy.zip"
}

Notice we parameterized role here with a variable defined in root level’s variables.tf file. We could parameterize the other variables as well, but for demo purpose, let’s just show one for now. The actual value of lambda_iam_role_arn is defined in terraform.tfvars file located under .env/dev folder.

lambda_iam_role_arn = "arn:aws:iam::##########:role/service-role/lambda-test-role-#####"

A better approach would be to use Terraformer to import code for that IAM role, and modularize that iam module, then in root main.tf, create a new Lambda IAM role, then use the output of calling the iam module in the Lambda block, role = module.iam.iam_role_arn, instead of using hard-coded variable from terraform.tfvars. Well, readers are welcome to use this scenario as an exercise to practice on your own.

Step 5: Add a GitHub Actions Workflow

Now with the Terraform project structure in place, let’s explore how to deploy our Terraform code to AWS with GitHub Actions workflow. See below a sample workflow to run the series of Terraform commands: terraform init, terraform plan, terraform apply, and finally if needed we can switch to a branch named destroy to run terraform destroy to erase the resources we just created in AWS.

name: "Terraform Deployment"

on:
  # Manual trigger
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment to run the workflow against'
        type: environment
        required: true
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

defaults:
  run:
    shell: bash

jobs:
  terraform:
    name:   Deploy terraform
    runs-on: ubuntu-latest

    # important to specify the environment here so workflow knows where to deploy your artifact to.
    # default environment to "dev" if it is not passed in through workflow_dispatch manual trigger
    environment: ${{ github.event.inputs.environment || 'dev' }}

    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}

    steps:
      - name: Harden Runner
        uses: step-security/harden-runner@74b568e8591fbb3115c70f3436a0c6b0909a8504
        with:
          egress-policy: audit # TODO: change to 'egress-policy: block' after couple of runs

      - name: Checkout
        uses: actions/checkout@d0651293c4a5a52e711f25b41b05b2212f385d28

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@7b3bcd8d76f3cbaec0a3564e53de7c9adf00f0a7

      - name: Terraform Init
        id: init
        run: |
          rm -rf .terraform
          terraform init -upgrade=true -no-color -input=false
      - name: Terraform Plan
        id: plan
        run: |
          terraform plan -input=false -var-file=.env/${{ github.event.inputs.environment || 'dev' }}/terraform.tfvars -no-color
      - name: Terraform Apply
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        id: apply
        run: terraform apply -auto-approve -input=false -var-file=.env/${{ github.event.inputs.environment || 'dev' }}/terraform.tfvars

      - name: Terraform destroy
        # If you want to use this workflow to run terraform destroy, create a feature branch "destroy", trigger this workflow from that branch to destroy.
        if: github.ref == 'refs/heads/destroy'
        id: destroy
        run: terraform destroy -auto-approve -input=false -var-file=.env/${{ github.event.inputs.environment || 'dev' }}/terraform.tfvars

Notice we are passing in terraform.tfvars with -var-file to the Terraform commands. Depending on the environment passed in from workflow_dispatch manual trigger, Terraform workflow checks in the environment folder under .env, runs for that chosen environment with the respective .tfvars file passed in.

Summary

We explored what Terraformer is, why we need it, and how to use it to import existing infrastructure into Terraform code. We then took a closer look at how to templatize the generated Terraform code and make it reusable. We also looked into some of the best practices for Terraform modular structure. Last but not least, we explored using GitHub Actions workflow to automate Terraform code deployment.

Happy coding!

Standard Module Structure | Terraform by HashiCorp

Originally published at https://betterprogramming.pub on August 24, 2022.