PI #006: Automate Your AWS Cloud Infrastructure Using Terraform
Terraform 101: How to Use Terraform as IaC to Automate the Creation & Deletion of Your AWS Infrastructure
This newsletter aims to give you weekly insights about designing and productionizing ML systems using MLOps good practices 🔥.
This week I want to show you how to use Terraform to automate the creation & deletion of your AWS infrastructure.
The one tool that ML engineers underestimate...
That will x10 your productivity.
Terraform Introduction
Using Terraform, you can define a blueprint of your entire infrastructure using Terraform and create or destroy it using just a few commands.
No more spending countless hours creating your EC2 instances manually one by one.
Here is a simple example of how to create an AWS EC2 instance using Terraform:
1. Install the provider (e.g., AWS, GCP, Azure, etc.) & set up the credentials:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "4.67.0"
}
}
}
provider "aws" {
region = "eu-central-1"
access_key = "<your_access_key>"
secret_key = "<your_secret_key>"
}
2. Define the resource you want to create - an AWS EC2 instance:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "4.67.0"
}
}
}
provider "aws" {
region = "eu-central-1"
access_key = "<your_access_key>"
secret_key = "<your_secret_key>"
}
resource "aws_instance" "my_terraformed_instance" {
ami = "ami-0ab1a82de7ca5889c"
instance_type = "t2.micro"
}
3. Initialize & apply the changes:
# Download providers and
# initialize the lock file.
terraform init
# [Optional] Print the execution
# plan based on your Terraform file.
terraform plan
# Create the resources based on
# your Terraform file.
terraform apply
# Done. Your AWS resources
# have been created.
If you run “terraform apply” again, as you haven't modified your Terraform file, it won't create additional resources.
Your Terraform file acts like a blueprint.
It is not like running a stateless script. Terraform saves the state.
Thus, your AWS infrastructure will always match 1:1 your infrastructure defined in your Terraform files.
4. Update your Terraform file:
Note the tags section with the “aws_instance” resource.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "4.67.0"
}
}
}
provider "aws" {
region = "eu-central-1"
access_key = "<your_access_key>"
secret_key = "<your_secret_key>"
}
resource "aws_instance" "my_terraformed_instance" {
ami = "ami-0ab1a82de7ca5889c"
instance_type = "t2.micro"
tags = {
Name = "terraformed-ubuntu"
}
}
5. Update your infrastructure. Delete your infrastructure:
# Update EC2 resource with the new tag.
terraform apply
# Destroy everything.
terraform destroy
# Destroy only a specific resource
terraform destroy -target aws_instance.my_terraformed_instance
# Create only a specific resource.
terraform apply -target aws_instance.my_terraformed_instance
Now Let’s Build a Production-Ready AWS Infrastructure
Now we will show you how to build a production-ready AWS EC2 infrastructure will all its components, such as networking, security groups, and EC2 configuration.
#1. Define the provider:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "4.67.0"
}
}
}
provider "aws" {
region = "eu-central-1"
access_key = "<your_access_key>"
secret_key = "<your_secret_key>"
}
#2. Create a VPC:
resource "aws_vpc" "prod-vpc" {
cidr_block = "10.0.0.0/16"
tags = {
Name = "production"
}
#3. Create an Internet Gateway:
# Note how we referenced the VPC
# using the resource type ("aws_vpc"),
# the resource name ("prod-vpc"),
# and the id.
resource "aws_internet_gateway" "prod-gw" {
vpc_id = aws_vpc.prod-vpc.id
tags = {
Name = "production"
}
}
#4. Create a Route Table:
# Again, notice how we reference other resources
# within the Terraform file, using the
# <resource_type>.<resource_name>.id format.
resource "aws_route_table" "prod-route-table" {
vpc_id = aws_vpc.prod-vpc.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.prod-gw.id
}
route {
ipv6_cidr_block = "::/0"
gateway_id = aws_internet_gateway.prod-gw.id
}
tags = {
Name = "production"
}
}
#5. Create a Subnet:
resource "aws_subnet" "subnet-1" {
vpc_id = aws_vpc.prod-vpc.id
cidr_block = "10.0.1.0/24"
availability_zone = "eu-central-1a"
tags = {
Name = "production-subnet"
}
}
# 6. Associate the Subnet with the Route Table:
resource "aws_route_table_association" "a" {
subnet_id = aws_subnet.subnet-1.id
route_table_id = aws_route_table.prod-route-table.id
}
# 7. Create a Security Group:
# Security Group to allow web traffic for ports 443, 80, and 22.
resource "aws_security_group" "allow_web_sg" {
name = "allow_web"
description = "Allow WEB traffic"
vpc_id = aws_vpc.prod-vpc.id
ingress {
description = "HTTPS"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
description = "HTTP"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
description = "SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
# This means, allow all outbound traffic.
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "allow_web"
}
}
# 8. Create a network interface:
resource "aws_network_interface" "web-server-nic" {
subnet_id = aws_subnet.subnet-1.id
private_ips = ["10.0.1.50"]
security_groups = [aws_security_group.allow_web_sg.id]
}
# 9. Assign an Elastic IP - EIP to the network interface:
# Here is a gotcha regarding the order of the resources.
# To create an EIP, the Internet Gateway must already be created.
# You can bypass this by using the "depends_on" attribute.
resource "aws_eip" "one" {
vpc = true
network_interface = aws_network_interface.web-server-nic.id
associate_with_private_ip = "10.0.1.50"
# Reference the whole object, not just the id.
depends_on = [ aws_internet_gateway.prod-gw ]
}
# 10. Create the EC2 instance and initialize it with an apache2 server:
resource "aws_instance" "web-server-instance" {
ami = "ami-0ab1a82de7ca5889c"
instance_type = "t2.micro"
availability_zone = "eu-central-1a"
key_name = "<your-key-pair-name>"
network_interface {
device_index = 0
network_interface_id = aws_network_interface.web-server-nic.id
}
# Set of commands to run on start-up.
user_data = <<-EOF
#!/bin/bash
sudo apt update -y
sudo apt install apache2 -y
sudo systemctl start apache2
sudo bash -c 'echo "..." > /var/www/html/index.html'
sudo systemctl enable apache2
EOF
tags = {
Name = "web-server"
}
}
#11. Main commands you have to know:
# Initialize providers/dependencies.
terraform init
# Print plan.
terraform plan
# Create resources.
terraform apply
# Destroy resources.
terraform destroy
Thats it!
You can create & destroy an entire infrastructure with a click of a button.
Amazing, right? 🔥
Tips & Tricks:
- most of the attributes of the Terraform resources correspond to their cloud vendor equivalent
- you find verbose examples for every resource if you check out the docs
- you will never be worried about forgetting to shut down an EC2 instance that will bankrupt you
Master writing clean & modular Terraform files
Using this one simple technique:
Variables
You can write clean & modular Terraform files.
Let's take a look at how to:
- define a variable
- reference a variable
- assign a value to a variable
- use files to assign values to variables
Note that assigning values in Terraform is quite strange, as in your Terraform file, you define the structure and type of the variable. The value is assigned only on runtime.
#1. This is how you define a variable in Terraform:
# You can provide the 3 following attributes to any variable,
# but not all 3 are mandatory:
variable "<name>" {
description = "<some_description>"
default = "<default_value>"
type = "<type_constraint>"
}
#2. Now let’s use it:
variable "subnet_prefix" {
description = "cidr block for the subnet"
}
resource "aws_subnet" "subnet-1" {
vpc_id = aws_vpc.prod-vpc.id
# Reference the variable using
# the following format:
# var.<variable_name>.
cidr_block = var.subnet_prefix
availability_zone = "eu-central-1a"
tags = {
Name = "production-subnet"
}
}
But wait...
So far, the value of the variable isn't specified anywhere.
When running “terraform apply”, you will be prompted to introduce the value of every variable.
Run “terraform apply” and when asked, introduce: “10.0.1.0/24”
#3. Variable files:
Do you see any issues so far?
What happens when you have 20 variables?
Do manually introduce them isn't a viable option...
That is why you can use Terraform .tfvars files.
Create a file called “terraform.tfvars”.
Add the following content to it:
subnet_prefix = "10.0.100.0/24"
By default, Terraform will always look for a file called “terraform.tfvars” and load the variables from it.
To explicitly say what file to use, run:
terraform apply -var-file example.tfvars
Note: You can also use environment variables to inject sensitive variables, such as credentials.
#4. Variable default values & types:
1. You can leverage the default value to add default settings but still leave room for customization.
2. You can also leverage type constraints to the variable to ensure that the right type is passed.
variable "subnet_prefix" {
description = "cidr block for the subnet"
default = "10.0.102.0/24" #1.
type = string #2.
}
See you next week on Thursday at 9:00 am CET.
Have a fabulous weekend!
💡 My goal is to help machine learning engineers level up in designing and productionizing ML systems. Follow me on LinkedIn and Medium for more insights!
🔥 If you enjoy reading articles like this and wish to support my writing, consider becoming a Medium member. Using my referral link, you can support me without extra cost while enjoying limitless access to Medium's rich collection of stories.
Thank you ✌🏼 !