Terraform Learning

 

• Context
• Please note that the following notes has been taken from the Zeal Vora Udemy course and from reddit answers.
• This blog just I have created for learning/Revision purpose.

• Terraform has wide range of providers available.
• AWS
• GCP
• Azure
  
• K8s
• Jenkins, etc.
• Provider is a plugin that enables terraform to interact with an API to manage resources.
• Browse Providers | Terraform Registry
• Terraform requires explicit source information for any providers that are not HashiCorp-maintained, using a new syntax in the required_providers nested block inside the terraform configuration block.

• Terraform's primary function is to create, modify, and destroy infrastructure resources to match the desired state described in a Terraform configuration whereas Current state is the actual state of a resource that is currently deployed.
• It could be possible that someone has manually changed the infrastructure so whenever you try to run terraform plan command it will try to get the desired state(The state that you have defined in your terraform file) by modifying current state(The state that is present in cloud).
• If you change the security group of instances manually and run terraform refresh command, there won't be any impact on terraform plan command unless you have defined security group explicitly in ec2 instance resource. and hence the changed security group will get added to terraform state file.
• terraform -refresh-only
• Since terraform refresh command is automatically inbuilt with terraform plan and terraform apply command hence you won't need to use it explicitly also sometimes it might create issue if you accidently changed the region, it will empty the state file.

Provider Architecture

• For production environment it is always recommended to provide provider version, it ensures that new versions with breaking changes will not be automatically installed.
• ~> 3.0 represent any version in the 3.X range.

• Docs overview | hashicorp/aws | Terraform | Terraform Registry
• Because of the lock file you would not be able to work with any other version, if you want to upgrade your provider version you could use the following command:

$ terraform init -upgrade

• Instead of hardcoding backend of terraform you could essentially pass the value with terraform init command, Something like following:

terraform init -reconfigure \
-backend-config="backet=$TERRAFORM_STATE_BUCKET"  \
-backend-config="key=$TERRAFORM_STATE_BUCKET_KEY" \
-backend-config="region"=$REGION  \
-backend-config="dynamodb_table=$DYNAMODB_TABLE" \
-input=false

• Terraform best practices
• String interpolation: ${...} This syntax indicates that terraform will replace the expression inside the curly braces with its calculated value. Also, it solves cross referencing issues.

resource "aws_eip" "lb" {
   domain  =  "vpc"
}

resource "aws_security_group" "example" {
   name = "attribute-sg"
}


resource "aws_vpc_security_group_ingress_rule" "example" {
   security_group_id   =  aws_security_group.example.id
   cidr_ipv4           =  "${aws_eip.lb.public_ip}/32"
   from_port           = 443
   ip_protocol         = "tcp"
   to_port             = 443   

}

• Variable precedence: Later source taking precedence over earlier one.

1. Environment variables
2. The terraform.tfvars file, if present.
3. The terraform.tfvars.json file, if present.
4. Any *.auto.tfvars or *.auto.tfvars.json files, processed in lexical order of their filenames.
5. Any -var and -var-file options on the command line

• It is considered a good practice to provide type of variable while defining it.
• To destroy a particular resource you could use terraform destroy command with -target flag
• terraform destroy -target aws_instance.myec2
• Fetching data from Maps & List in variable

variable "instance_type" {
    type = map
    default = {
       us-east-1 = "t2.micro"
       us-west-1 = "t3.micro"
    }
}

variable "instance_types" {
     type = list
     default = ["t2.small", "t3.medium"]
}
instance_type = var.instance_type["us-east-1"]

instance_type = var.instance_types[0]



• Count Index: basically, to configure resources which are getting generated as part of count variable.
• Sometimes, you may not want loadbalancer0 as naming convention hence you could define the name in list and fetch it through loop.

resource "aws_iam_user" "lb" {
   name  = "loadbalancer.${count.index}"
   count = 2
   path  = "/system/"
}


variable "elb_names" {
   type = list
   default = ["dev-loadbalancer", "stage-loadbalancer"]
}
-------------------------------------------------------------------------------
resource "aws_iam_user" "lb" {
   name  = var.elb_names[count.index]
   count = 2
   path  = "/system/"
}


variable "elb_names" {
   type = list
   default = ["dev-loadbalancer", "stage-loadbalancer"]
}

• Conditional Expression: It uses the bool expression to select one of two values.

condition ? true_val : false_val

• A module is like a function, A module block is like a call to a function.
  
• An input variable is like a parameter to a function, The definition of an input variable with a particular value (inside a module block) is alike an argument passed to the function.
• A local value is like a local variable within a function, it has a local scope so it's not visible in any other module.
• local values can be used for multiple different use-cases like having a conditional expression
locals {
   name_prefix  = "${var.name != "" ? var.name: var.default}"
}


• You may use locals unless input variables are necessary.
• Input variables have limitations, while they can contain default values, those values can't be computed and don't support expressions.
• local variables fully support expressions, so when you have a particularly complex bit of expressing to do for a value, you can do it in local and ship the complexity away from the resource code.
• And an output is like a named return value from a function.
• Use of conditional expression in following terraform file root module & child module:

module "vpc" {  
   source                 = "./module/vpc"  
create_vpc             = true  
        Project                = var.Project  
        region                 = var.region  
vpc_cidr_block         = var.vpc_cidr    
tags = {    
  "Owner" = var.Owner,     
  "Email" = var.Email   
  }  
public_subnet_count    = 3  
public_cidr            = var.public_cidr  
azs                    = var.azs  
public_subnet_names    = var.public_subnet_names  
public_subnet_suffix   = ""  
create_igw             = true  
private_subnet_count   = 3  
private_cidr           = var.private_cidr  
private_subnet_names   = var.private_subnet_names  
private_subnet_suffix  = ""
}

data "aws_caller_identity" "current" {}

locals{
account_id = data.aws_caller_identity.current.account_id
}


locals { 
   create_vpc = var.create_vpc
}

resource "aws_vpc" "this" {
   cidr_block                           = var.vpc_cidr_block 
   enable_dns_hostnames                 = true 
   enable_dns_support                   = true 
   enable_network_address_usage_metrics = false
   tags = merge(
   {
      "Name" = var.Project },   
       var.tags 
   )   
   #checkov:skip=CKV2_AWS_11:flow logs are disable deliberately as of now.


####################### CREATE PUBLIC SUBNETS###########################
locals { 
   create_public_subnet = local.create_vpc && var.public_subnet_count > 0
}

resource "aws_subnet" "public" { 
  count         = local.create_public_subnet ? var.public_subnet_count :0
  vpc_id        = aws_vpc.this.id 
  cidr_block    = var.public_cidr[count.index] 
  availability_zone  = length(regexall("^[a-z]{2}-", element(var.azs,count.index))) > 0 ? element(var.azs, count.index) : null 
  tags = merge(   
  {     
     Name = try(       
       "${var.Project}-${var.public_subnet_names[count.index]}",                             format("${var.Project}-${var.public_subnet_suffix}-%s",
       element(var.azs, count.index))     
       )   
   },   
   var.tags 
  )
}


resource "aws_route_table" "public" { 
   count   = local.create_public_subnet ? 1 : 0 
   vpc_id  = aws_vpc.this.id  tags    = merge(   
   {
      "Name" = "${var.Project}-${var.public_subnet_suffix}-rt" },   
      var.tags 
    )
}


resource "aws_route_table_association" "public" { 
    count          = local.create_public_subnet ? var.public_subnet_count : 0              subnet_id      = element(aws_subnet.public[*].id, count.index) 
    route_table_id = aws_route_table.public[0].id
}


resource "aws_route" "public_internet_gateway" { 
    count                  = local.create_public_subnet && var.create_igw ? 1 : 0          route_table_id         = aws_route_table.public[0].id 
    destination_cidr_block = "0.0.0.0/0" 
    gateway_id             = aws_internet_gateway.this[0].id
}


resource "aws_internet_gateway" "this" { 
    count   = local.create_public_subnet && var.create_igw ? 1 : 0 
    vpc_id  = aws_vpc.this.id 
    tags    = merge(    { "Name" = var.Project },   
       var.tags  )
}

##################### CREATE PRIVATE SUBNET##########################

locals { 
   create_private_subnet = local.create_vpc && var.private_subnet_count > 0
}


resource "aws_subnet" "private" { 
   count             = local.create_private_subnet ? var.private_subnet_count : 0         vpc_id            = aws_vpc.this.id 
   cidr_block        = var.private_cidr[count.index] 
   availability_zone = length(regexall("^[a-z]{2}-", element(var.azs, count.index))) > 0 ? element(var.azs, count.index) : null 

   tags = merge(    {     
        Name = try(        "${var.Project}-${var.private_subnet_names[count.index]}",          format("${var.Project}-${var.private_subnet_suffix}-%s", element(var.azs,             count.index))     
   )},   
   var.tags  )
}


resource "aws_route_table" "private" { 
    count   = local.create_private_subnet  ? 1 : 0 
    vpc_id  = aws_vpc.this.id 
    tags    = merge(   
    {
        "Name" = "${var.Project}-${var.private_subnet_suffix}-rt"   
    },    var.tags  )
}


resource "aws_route_table_association" "private" { 
     count          = local.create_private_subnet ? var.private_subnet_count : 0           subnet_id      = element(aws_subnet.private[*].id, count.index)  route_table_id = aws_route_table.private[0].id
}


resource "aws_default_security_group" "default" { 
     vpc_id = aws_vpc.this.id
}   

• Terraform functions

## lookup syntax
lookup(map, key, default)


locals {
  time = formatdate("DD MMM YYYY hh:mm ZZZ", timestamp())
}

variable "ami" {
  type = map
  default = {
    "us-east-1" = "ami-0323c3dd2da7fb37d"
    "us-west-2" = "ami-0d6621c01e8c2de2c"
    "ap-south-1" = "ami-0470e33cd681b2476"
  }
}

variable "region" {
  default = "ap-south-1"
}

resource "aws_instance" "app-dev" {
   ami = lookup(var.ami,var.region)
}



# element function
variables "tags"{
    type = list
    default = ["firstec2","secondec2"]
}

resource "aws_instance" "app-dev" {
   instance_type = "t2.micro"   
    tags = {
       Name = element(var.tags, cound.index)
   }
}


• data sources in terraform. ref
• data sources allow terraform to use/fetch information defined outside of terraform
• data sources allow data to be fetched from provider to use in terraform configuration.
• Terraform data sources let you dynamically fetch data from APIs or other Terraform state backends. Examples of data sources include machine image IDs from a cloud provider or Terraform outputs from other configurations.
• Following data source allows you to read contents of a file in your local filesystem
• data source can be used to fetch information about a specific IAM role. By using this data source, you can reference IAM role properties without having to hard code ARNs as input.

data "aws_iam_role" "example" {
   name = "an_example_role_name"
}


data "local_file" "foo" {
   filename = "${path.module}/demo.txt"
}

## ${path.module} returns the current filesystem path where code is located

## TO fetch the ami, it also supports filter

data "aws_ami" "web" {
    filter {
       name    =   "state"
       values  =   ["available"]
    }

    filter {
       name   =  "tag:Component"
       values = ["web"]
    }

    most_recent = true
}

• terraform validate && terraform fmt
• fmt helps to format the tf code
• whereas validate primarily checks whether a configuration is syntactically valid.
• Dynamic block in terraform
• Dynamic block allows us to dynamically construct repeatable nested blocks which is supported inside resource, data, provider and provisioner blocks.
• Dynamic block vs for_each block
• for_each: Use it when you want to create multiple resources (like EC2 instances, S3 buckets, etc.).
• dynamic block: Use it when you want to create multiple nested blocks inside a single resource, like multiple ingress rules in a security group.
• Count: When looping a fixed number of items

variable "sg_ports" {
  type        = list(number)
  description = "list of ingress ports"
  default     = [8200, 8201,8300, 9200, 9500]
}

resource "aws_security_group" "dynamicsg" {
  name        = "dynamic-sg"
  description = "Ingress for Vault"

  dynamic "ingress" {
    for_each = var.sg_ports
    iterator = port
    content {
        from_port   = port.value
        to_port     = port.value
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
  }

  dynamic "egress" {
     for_each = var.sg_ports
     content {
        from_port   = egress.value
        to_port     = egress.value
        protocol    = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
     }
  }
}
##creation of subnets
variable "subnet_names" {
    type    =   list(string)
    default =   ["subnet-pub01","subnet-pub02"]
}

variable "subnet_cidr" {
    type    = list(string)
    default = ["10.0.0.0/20","10.0.16.0/20"]
}

dynamic "subnet"{
   for_each  =  zipmap(var.subnet_names,var.subnet_cidr)
   content   = {
      name           = subnet.key
      address_prefix = subnet.value
   }
 }
}

• Terraform taint
• When a resource's actual configuration drifts from what's defined in Terraform.
• When a resource needs to be recreated due to the fact that either resources are corrupted or not working as expected.
• It marks the resource and create/destroy it on next apply.
• terraform taint is deprecated hence recommend using replace command.

$ terraform apply -replace="aws_instance.myec2"



• splat expression

## following is the output of VPC module
output "public_subnet_ids" {
   value = aws_subnet.public[*].id
}


## I am trying to fetch the subnet ids into lambda module
moduel "lambda" {
   lambda_subnet_ids = module.vpc.public_subnet_ids
}

• How to ignore manual changes from AWS console:
• Let's say you created a ec2 instance you added a Name tag however someone from your team added a Tag ENV as Production so with the default behavior of terraform on the next terraform apply command run terraform will remove the Tag.
• You could use Meta Arguments to ignore the changes

resource "aws_instance" "myec2" {
   ami = "ami-67687690709709"
   instance_type = "t2.micro"

   lifecycle{
      ignore_changes = [tags] 
  }

}


## Different Meta Arguments

• depends_on
• count
• for_each
• lifecycle
• provider

## lifecycle is a nested block that can appear within a resource block. The lifecycle block and its contents are meta-arguments, available for all resource blocks regardless of type. The arguments available within a lifecycle block are:

• create_before_destroy
• prevent_destroy
• ignore_changes
• replace_triggered_by


## In production env you may want to create resource before destroying it, basically to override default property of terraform

lifecycle{
   create_before_destroy = true
}


lifecycle{
   prevent_destroy = true
}


resource "aws_instance" "myec2"{
    ami           =  "ami-0979dikjkfdfef"
    instance_type =  "t2.micro"
    tags = {
       Name = "Web Server"
    }
    lifecycle {
       ignore_changes = [tags]
    }
}

• Challenges with count.index in terraform:
• things may get messy if you try to increase the count in future and you couldn't put the value in list properly.
• so if you resources are almost identical, count is appropriate, and if distinctive values are needed in the arguments, go for for_each
• Let's say if you need to create identical 5 ec2 resources with same instance type go for count and if the instance type are different then go for for_each

resource "aws_iam_user" "iam" {
   for_each = toset( ["user-01","user-02","user-03"])
   name = each.key
}


resource "aws_instance" "myec2" {
   ami = "ami-8786786786876"
   for_each  =  {
       key1  =  "t2.micro"
       key2  =  "t3.micro"
   }
   instance_type  =  each.value
   key_name       =  each.key

   tags = {
      Name = each.value
  }
}

• Provisioners
• Provisioners are used to execute scripts on a local or remote machine as part of resource creation or destruction, for example - after VM is launched, install software package required for application
• Provisioners are defined inside a specific resource, for remote exec you need to define connection as well.
• Provisioner: file | Terraform | HashiCorp Developer
• There are two major types of provisioners available 
• local-exec : The local-exec provisioner invokes a local executable after a resource is created, example : After EC2 is launched, fetch the IP and store in file server_ip.txt
• remote-exec : remote-exec provisioners allow to invoke scripts or run commands directly on the remote server.
• file is also a provisioner but it is not widely used
• Most provisioners require access to the remote resource via SSH or WinRM and expect a nested connection block with details about how to connect.
• 
  
  
  
  

• If when =  destroy is specified, the provisioner will run when the resource it is defined within is destroyed

resource "aws_instance" "web" {
   ....
  provisioner "local-exec" {
   when = destroy OR creation
   command = "echo 'Destroy-time provisioner'"
  }
}



• Provisioner Failure Behavior
• By default, provisioners that fail will also cause the Terraform apply itself to fail
• The on_failure setting can be used to change this.

• DRY Approach(Don't repeat yourself)
• local block use case
• Let's say in the child module you have defined an application port for the ingress rule of security group and since there are multiples rule with same port you may want to define that port to variable, defining variable may lead to get overridden from root module hence in child module you could use local block in order to define that variable as local variable.
• If you want the output of any resource's attribute you would have to define output in root module
• You could use Terraform Registry to browse the module and utilize it in your terraform resources.

• Terraform workspace
• You could manage different environments using terraform workspace
• To create new workspace run the following command

$ terraform workspace -h
$ terraform workspace show
$ terraform workspace new dev
$ terraform workspace new prd
$ terraform workspace list
$ terraform workspace select dev

• Configure workspace variable according to environments

resource "aws_instance" "myec2" {
   ami = "ami-082b5a644766e0e6f"
   instance_type = lookup(var.instance_type,terraform.workspace)
}
variable "instance_type" {
  type = "map"

  default = {
    default = "t2.nano"
    dev     = "t2.micro"
    prd     = "t2.large"
  }

}

• Module source
• Source of module could be local folder
• git protocol
• and many more
• Terraform state management.

# The Terraform state list command is used to list resources within a Terraform state
$ terraform state list

resource "aws_instance" "myec2" {
   ami = "ami-082b5a644766e0e6f"
   instance_type = "t2.micro"
}

## you want following you could use terraform state mv command

resource "aws_instance" "mywebapp" {
   ami = "ami-082b5a644766e0e6f"
   instance_type = "t2.micro"
}


# The Terraform state mv command is used to move items in a Terraform state
# Basically, if you want to rename an existing resource without destroying and recreating it 
$ terraform state mv aws_instance.webapp aws_instance.myec2


# Terraform state pull command is used to manually download and output the state from remote state.
# This is useful for reading values out of state (you could use jq)
$terraform state pull

## if you want to remove items from the terraform state, however it will be available in AWS but terraform will remove from its states
$ terraform state rm aws_instance.myec2 

Terraform remote state management:

Let's say there are two projects A & B, how could you use Project's A output (A's terraform state file is in S3 bucket) in Project B

Name of the state file is :  eip.tfstate

# Project A
output "eip_addr" {
   value = aws_eip.lb.public_ip
}
$cat s3://satish-terraform-backend/network/eip.tfstate
{
  "outputs": {
     "eip_addr" : {
        "value": "54.90.4.33",
        "type" : "string"
  }
 }
}

# Project B
$cat remote-state.tf
data "terraform_remote_state" "eip" {
    backend = "s3"
    config = {
        bucket = "satish-terraform-backend"
        key    = "network/eip.tfstate"
        region = "us-east-1"
   }
}
resource "aws_security_group" "allow_tls" {
   
  ingress{
     cidr_blocks  = ["${data.terraform_remote_state.eip.outputs.eip_addr}/32"]
  }
}

• Terraform import
• Terraform import can automatically create the terraform configuration files for the resources you want to import.

provider "aws" {
   region = "us-east-1"
}

import {
   to = aws_security_group.mysg
   id = "sg-9u897r9889r9r" 
}

$ terraform plan -generate-config-out=mysg.tf
$ terraform apply -auto-approve

• Single provider multiple configuration
• You could use Alias in order to launch the resource in multiple region

provider "aws" {
  region = "us-west-1"
}

provider "aws" {
   alias  = "mumbai"
   region = "ap-south-1"
}


resource "aws_eip" "myeip" {
  vpc = true
}

resource "aws_eip" "myeip01" {
   vpc      = true
   provider = "aws.mumbai" 
}

#To work with multiple account essentially you could just add profile to provider block

provider "aws" {
  region = "us-west-1"
}

provider "aws" {
   alias  = "mumbai"
   region = "ap-south-1"
   profile = account2
}

• Assume role with provider.

provider "aws"{
   region = "us-east-1"
   assume_role{
      role_arn     = "arn:aws:iam:85858868584:role/sysadminrole"
      session_name = "test" 
   }
}

• Sensitive Parameter

output "db_password" {
   value       = aws_db_instance.db.password
   description = "The password for logging in to the database"
   sensitive   = true
}

• AWS Provider - Authentication configuration

provider "aws" {
    shared_config_files      =  ["/users/tf_user/.aws/config"]
    shared_credentials_files = ["/users/tf_user/.aws/creds"]
    profile                  = "customprofile"
}

Challenges with Larger Infrastructure

• When you have a larger infrastructure, you will face issue related to API limits for a provider
• You can switch to smaller configuration where each be applied independently for example writing multiple resources to a single file may not be a good idea, instead what you can do it basically create multiple resource based tf file.
• You can prevent terraform from quering the current state during operations like terraform plan
• terraform plan -refresh=false
• You could specify the target
• terraform plan -target=ec2

 

• Use terraform visual studio extension in order to get the help from syntax and auto completion point of view:

Terraform functions

• The zipmap function constructs a map from a list of keys and a corresponding list of values
• element function retrieves a single element from a list
• lookup function retrieves the value of a single element from a map

zipmap(["a","b"],[1,2])
{
  "a"  = 1
  "b"  = 2
}

gitignore

• conf/
• *.artifacts
• credentials
• .terraform
• terraform.tfvars
• terraform.tfstate
• crash.log

Thanks for reading!!

Reference Links:

• GitHub - evairmarinho/terraform-levels: Terraform examples in levels.