개별 VPC 생성

VPC 및 Subnet 구성

VPC 구성

VPC 구성은 다른 AWS 리소스를 생성하기 전에 미리 세팅해주는 것이 좋습니다. 본 실습에서는 dayone-prod 에 dev와 prod환경 VPC를 구성해보도록 하겠습니다.

실제 서비스 운영을 위해서는 Developer 계정을 따로 만드는 것이 좋습니다. id 계정은 사용자가 콘솔에 접속하기 위해서만 사용하는 것이 권한 관리에 적합합니다.

실습에 앞서, 아래 과정은 개별적으로 진행 부탁드립니다. backend.tf 파일 수정은 앞선 IAM 세팅을 참고하시기 바랍니다.

$ cd terraform/vpc/dayoned_apnortheast2
$ vim backend.tf ---> 수정
$ terraform init

// Assume role to account
$ ./terraform_setup.sh --setup -p | pbcopy

## paste!! 
$ export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_SESSION_TOKEN="..."

Dev 환경 구성

Security Group 생성

  • VPC에서 공통으로 사용할 Security Group을 생성합니다.

  • 사무실/집에서 SSH 접속을 위한 Security Group과 Private Subnet에 있는 인스턴스에 접속하기 위한 Bastion 서버의 Security Group을 생성할 예정입니다.

  • Security Group ID는 추후에 다른 리소스 생성에 필요하므로 Output으로 빼야 합니다.

  • 공통으로 필요한 Security Group이 있으면 본 실습에서 추가로 생성하시면 됩니다.

Default/Home Security Group

  • 10.0.0.0/8 is the CIDR range we are going to use for all VPC setting

  • Default SG : 인스턴스가 공통으로 가져야 할 보안그룹입니다.

  • Home SG : Admin 페이지, kibana 페이지 등 접근 제어가 필요한 웹사이트에 회사에서만 접속할 수 있도록 설정하기 위한 보안그룹입니다.

아래 예시 중에서 Node Exporter, Jmx Exporter, Kafka, Elasticsearch에 대한 inbound/outbound rule은 사용법을 알려드리기 위해 추가한 것입니다. 필요한 설정으로 변경 또는 삭제 후에 apply하시기 바랍니다.

vim terraform/vpc/dayoned_apnortheast2/default_sg.tf
# Default Security Group 
# This is the security group for most of instances should have 
resource "aws_security_group" "default" {
  name        = "default-${var.vpc_name}"
  description = "default group for ${var.vpc_name}"
  vpc_id      = aws_vpc.default.id

  #ingress {
  #  from_port = 80       # You could set additional ingress port 
  #  to_port   = 80
  #  protocol  = "tcp"

  #  cidr_blocks = [
  #    "10.0.0.0/8",
  #  ]
  #}
    
  # Instance should allow jmx exportor to access for monitoring
  ingress {
    from_port   = 10080
    to_port     = 10080
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/8"] 
    description = "inbound rule for jmx exporter"
  }


  # Instance should allow node exporter to access for monitoring
  ingress {
    from_port   = 9100
    to_port     = 9100
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/8"]
    description = "inbound rule for node exporter"
  }

  egress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "https any outbound"
  }

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "https any outbound"
  }

  # Instance should allow ifselt to send the log file to kafka
  egress {
    from_port   = 9092
    to_port     = 9092
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "kafka any outbound"
  }


  # Instance should allow ifselt to send the log file to elasticsearch
  egress {
    from_port   = 9200
    to_port     = 9200
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "ElasticSearch any outbound"
  }

}

# Home Security Group 
# This will be usually attached to the web server that users in the office should access through browser..
# This is used for all users in the company to access to the resources in the office or home..
resource "aws_security_group" "home" {
  name        = "home"
  description = "Home Security Group for ${var.vpc_name}"
  vpc_id      = aws_vpc.default.id

  ingress {
    from_port = 22
    to_port   = 22
    protocol  = "tcp"

    cidr_blocks = [
       "0.0.0.0/0"  # Change here to your office or house ...
    ]
  }

  ingress {
    from_port = 443
    to_port   = 443
    protocol  = "tcp"

    cidr_blocks = [
       "0.0.0.0/0"  # Change here to your office or house ...
    ]
  }

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "https any outbound"
  }
}

Bastion Security Group (Optional)

  • Bastion SG : Bastion 서버에 사용할 보안그룹입니다.

  • Bastion Aware SG : Bastion을 통해 접속을 할 수 있도록 권한을 허용해주는 보안그룹입니다. 이 보안그룹은 private 인스턴스에 붙일 예정입니다.

Bastion 서버 없이 AWS Session Manager, Teleport 등을 통해서 SSH 접속을 하는 경우에는 아래 보안그룹은 생성하실 필요가 없습니다.

vim terraform/vpc/dayoned_apnortheast2/bastion_sg.tf
# Security Group to the bastion server
resource "aws_security_group" "bastion" {
  name        = "bastion-${var.vpc_name}"
  description = "Allows SSH access to the bastion server"

  vpc_id = aws_vpc.default.id

  ingress {
    from_port = 22        # Specify the port you use for SSH
    to_port   = 22
    protocol  = "tcp"

    cidr_blocks = [
      "0.0.0.0/0"  # Change here to your office or house ...
    ]
  }

  egress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "http port any outbound"
  }

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "https port any outbound"
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["10.0.0.0/8"]
  }

  tags = {
    Name = "bastion-${var.vpc_name}"
  }
}

# Security Group from the bastion server
# This will be attached to the private instance which user wants to access through bastion host
resource "aws_security_group" "bastion_aware" {
  name        = "bastion_aware-${var.vpc_name}"
  description = "Allows SSH access from the Bastion server"

  vpc_id = aws_vpc.default.id

  ingress {
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    security_groups = [aws_security_group.bastion.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "bastionAware-${var.vpc_name}"
  }
}

Route53 Record

VPC에서 사용할 Internal DNS Record를 생성합니다. 내부통신에서 DNS를 사용하기 위해서 VPC마다 생성하는 것이 좋습니다.

VPC마다 독립적이기 때문에 이름은 중복되도 괜찮습니다. 본 실습에서는 모든 VPC의 내부 DNS는 dayone.internal 로 통일합니다.

vim terraform/vpc/dayoned_apnortheast2/route53.tf
resource "aws_route53_zone" "internal" {
  name    = "dayone.internal"
  comment = "${var.vpc_name} - Managed by Terraform"

  vpc {
    vpc_id = aws_vpc.default.id
  }
}

VPC 세트 생성

VPC에 필요한 기본 구성들을 패키지로 설치합니다. 본 파일에는 아래의 구성요소들이 들어있습니다.

  • VPC

  • Public Subnet / Private Subnet / DB Subnet(private)

  • Elastic IP for NAT

  • Route Table

  • Internet Gateway

  • NAT Gateway

vim terraform/vpc/dayoned_apnortheast2/vpc.tf
# VPC
# Whole network cidr will be 10.0.0.0/8 
# A VPC cidr will use the B class with 10.xxx.0.0/16
# You should set cidr advertently because if the number of VPC get larger then the ip range could be in shortage.
resource "aws_vpc" "default" {
  cidr_block           = "10.${var.cidr_numeral}.0.0/16" # Please set this according to your company size
  enable_dns_hostnames = true

  tags = {
    Name = "vpc-${var.vpc_name}"
  }
}

# Internet Gateway
resource "aws_internet_gateway" "default" {
  vpc_id = aws_vpc.default.id

  tags = {
    Name = "igw-${var.vpc_name}"
  }
}


# NAT Gateway 
resource "aws_nat_gateway" "nat" {
  # Count means how many you want to create the same resource
  # This will be generated with array format
  # For example, if the number of availability zone is three, then nat[0], nat[1], nat[2] will be created.
  # If you want to create each resource with independent name, then you have to copy the same code and modify some code
  count = length(var.availability_zones)

  # element is used for select the resource from the array 
  # Usage = element (array, index) => equals array[index]
  allocation_id = element(aws_eip.nat.*.id, count.index)
  
  #Subnet Setting
  # nat[0] will be attached to subnet[0]. Same to all index.
  subnet_id = element(aws_subnet.public.*.id, count.index)

  lifecycle {
    create_before_destroy = true
  }

  tags = {
    Name = "NAT-GW${count.index}-${var.vpc_name}"
  }

}

# Elastic IP for NAT Gateway 
resource "aws_eip" "nat" {
  # Count value should be same with that of aws_nat_gateway because all nat will get elastic ip
  count = length(var.availability_zones)
  vpc   = true

  lifecycle {
    create_before_destroy = true
  }
}


#### PUBLIC SUBNETS
# Subnet will use cidr with /20 -> The number of available IP is 4,096  (Including reserved ip from AWS)
resource "aws_subnet" "public" {
  count  = length(var.availability_zones)
  vpc_id = aws_vpc.default.id

  cidr_block              = "10.${var.cidr_numeral}.${var.cidr_numeral_public[count.index]}.0/20"
  availability_zone       = element(var.availability_zones, count.index)

  # Public IP will be assigned automatically when the instance is launch in the public subnet
  map_public_ip_on_launch = true

  tags = {
    Name = "public${count.index}-${var.vpc_name}"
  }
}

# Route Table for public subnets
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.default.id

  tags = {
    Name = "publicrt-${var.vpc_name}"
  }
}


# Route Table Association for public subnets
resource "aws_route_table_association" "public" {
  count          = length(var.availability_zones)
  subnet_id      = element(aws_subnet.public.*.id, count.index)
  route_table_id = aws_route_table.public.id
}




#### PRIVATE SUBNETS
# Subnet will use cidr with /20 -> The number of available IP is 4,096  (Including reserved ip from AWS)
resource "aws_subnet" "private" {
  count  = length(var.availability_zones)
  vpc_id = aws_vpc.default.id

  cidr_block        = "10.${var.cidr_numeral}.${var.cidr_numeral_private[count.index]}.0/20"
  availability_zone = element(var.availability_zones, count.index)

  tags = {
    Name               = "private${count.index}-${var.vpc_name}"
    immutable_metadata = "{ \"purpose\": \"internal_${var.vpc_name}\", \"target\": null }"
    Network            = "Private"
  }
}

# Route Table for private subnets
resource "aws_route_table" "private" {
  count  = length(var.availability_zones)
  vpc_id = aws_vpc.default.id

  tags = {
    Name    = "private${count.index}rt-${var.vpc_name}"
    Network = "Private"
  }
}

# Route Table Association for private subnets
resource "aws_route_table_association" "private" {
  count          = length(var.availability_zones)
  subnet_id      = element(aws_subnet.private.*.id, count.index)
  route_table_id = element(aws_route_table.private.*.id, count.index)
}


# DB PRIVATE SUBNETS
# This subnet is only for the database. 
# For security, it is better to assign ip range for database only.
# This is also going to use /20 cidr, which might be too many IPs... Please count it carefully and change the cidr.
resource "aws_subnet" "private_db" {
  count  = length(var.availability_zones)
  vpc_id = aws_vpc.default.id

  cidr_block        = "10.${var.cidr_numeral}.${var.cidr_numeral_private_db[count.index]}.0/20"
  availability_zone = element(var.availability_zones, count.index)

  tags = {
    Name               = "db-private${count.index}-${var.vpc_name}"
    immutable_metadata = "{ \"purpose\": \"internal_db_${var.vpc_name}\", \"target\": null }"
    Network            = "Private"
  }
}

# Route Table for DB subnets
resource "aws_route_table" "private_db" {
  count  = length(var.availability_zones)
  vpc_id = aws_vpc.default.id

  tags = {
    Name    = "privatedb${count.index}rt-${var.vpc_name}"
    Network = "Private"
  }
}

# Route Table Association for DB subnets
resource "aws_route_table_association" "private_db" {
  count          = length(var.availability_zones)
  subnet_id      = element(aws_subnet.private_db.*.id, count.index)
  route_table_id = element(aws_route_table.private_db.*.id, count.index)
}

Route Table Routes

이전에 만든 IGW와 NAT를 사용할 수 있도록 route table에 등록해야 합니다. 추후에 peering용으로 routes를 추가할 예정이므로 편의상 파일을 분리해서 관리합니다.

vim terraform/vpc/dayoned_apnortheast2/route_table_routes.tf
# routes for internet gateway which will be set in public subent
resource "aws_route" "public_internet_gateway" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.default.id
}

# routes for NAT gateway which will be set in private subent
resource "aws_route" "private_nat" {
  count                  = length(var.availability_zones)
  route_table_id         = element(aws_route_table.private.*.id, count.index)
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = element(aws_nat_gateway.nat.*.id, count.index)
}

Output

  • VPC에서 생성된 리소스는 추후에 다른 리소스에서 사용할 예정입니다.

  • 사용할 값들은 전부 output으로 추가해 놓습니다.

vim terraform/vpc/dayoned_apnortheast2/outputs.tf
# Region
output "aws_region" {
  description = "Region of VPC"
  value = var.aws_region
}

output "region_namespace" {
  description = "Region name without '-'"
  value = replace(var.aws_region, "-", "")
}

# Availability_zones
output "availability_zones" {
  description = "Availability zone list of VPC"
  value = var.availability_zones
}

# VPC
output "vpc_name" {
  description = "The name of the VPC which is also the environment name"
  value       = var.vpc_name
}

output "vpc_id" {
  description = "VPC ID of newly created VPC"
  value = aws_vpc.default.id
}

output "cidr_block" {
  description = "CIDR block of VPC"
  value = aws_vpc.default.cidr_block
}

output "cidr_numeral" {
  description = "number that specifies the vpc range (B class)"
  value = var.cidr_numeral
}

# Shard
output "shard_id" {
  description = "The shard ID which will be used to distinguish the env of resources"
  value       = var.shard_id
}

output "shard_short_id" {
  description = "Short version of shard ID"
  value       = var.shard_short_id
}

# Prviate subnets
output "private_subnets" {
  description = "List of private subnet ID in VPC"
  value = aws_subnet.private.*.id
}

# Public subnets
output "public_subnets" {
  description = "List of public subnet ID in VPC"
  value = aws_subnet.public.*.id
}

# Private Database Subnets
output "db_private_subnets" {
  description = "List of DB private subnet ID in VPC"
  value = aws_subnet.private_db.*.id
}

# Route53
output "route53_internal_zone_id" {
  description = "Internal Zone ID for VPC"
  value = aws_route53_zone.internal.zone_id
}

output "route53_internal_domain" {
  description = "Internal Domain Name for VPC"
  value = aws_route53_zone.internal.name
}

# Security Group
output "aws_security_group_bastion_id" {
  description = "ID of bastion security group"
  value = aws_security_group.bastion.id
}

output "aws_security_group_bastion_aware_id" {
  description = "ID of bastion aware security group"
  value = aws_security_group.bastion_aware.id
}

output "aws_security_group_default_id" {
  description = "ID of default security group"
  value = aws_security_group.default.id
}

output "aws_security_group_home_id" {
  description = "ID of home security group"
  value = aws_security_group.home.id
}

# ETC
output "env_suffix" {
  description = "Suffix of the environment"
  value       = var.env_suffix
}

output "billing_tag" {
  description = "The environment value for biliing consolidation."
  value       = var.billing_tag
}

리소스 생성

Terraform plan / apply 를 통해서 리소스를 생성합니다. 

$ terraform plan -parallelism=30
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_eip.nat[0] will be created
  + resource "aws_eip" "nat" {
      + allocation_id     = (known after apply)
      + association_id    = (known after apply)
      + domain            = (known after apply)
      + id                = (known after apply)
      + instance          = (known after apply)
      + network_interface = (known after apply)
      + private_dns       = (known after apply)
      + private_ip        = (known after apply)
      + public_dns        = (known after apply)
      + public_ip         = (known after apply)
      + public_ipv4_pool  = (known after apply)
      + vpc               = true
    }

  # aws_eip.nat[1] will be created
  + resource "aws_eip" "nat" {
      + allocation_id     = (known after apply)
      + association_id    = (known after apply)
      + domain            = (known after apply)
      + id                = (known after apply)
      + instance          = (known after apply)
      + network_interface = (known after apply)
      + private_dns       = (known after apply)
      + private_ip        = (known after apply)
      + public_dns        = (known after apply)
      + public_ip         = (known after apply)
      + public_ipv4_pool  = (known after apply)
      + vpc               = true
    }

  # aws_eip.nat[2] will be created
  + resource "aws_eip" "nat" {
      + allocation_id     = (known after apply)
      + association_id    = (known after apply)
      + domain            = (known after apply)
      + id                = (known after apply)
      + instance          = (known after apply)
      + network_interface = (known after apply)
      + private_dns       = (known after apply)
      + private_ip        = (known after apply)
      + public_dns        = (known after apply)
      + public_ip         = (known after apply)
      + public_ipv4_pool  = (known after apply)
      + vpc               = true
    }
    
( ... 생략 ... )
 

Plan: 42 to add, 0 to change, 0 to destroy.

Plan: 42 to add, 0 to change, 0 to destroy. 가 나오면 정상입니다. (수정한 리소스가 있는 경우 결과가 다를 수 있습니다.)

이제, 생성을 진행합니다.

$ terraform apply -parallelism=30

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:
(... plan 결과 생략 ...)

Plan: 42 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes
  
---------------------------------------------------------------------

Apply complete! Resources: 42 added, 0 changed, 0 destroyed.

Outputs:

availability_zones = [
  "ap-northeast-2a",
  "ap-northeast-2c",
  "ap-northeast-2b",
]
aws_region = ap-northeast-2
aws_security_group_bastion_aware_id = sg-xxxxxxx
aws_security_group_bastion_id = sg-yyyyyyy
aws_security_group_default_id = sg-zzzzzzz
aws_security_group_home_id = sg-aaaaaaa
billing_tag = dev
cidr_block = 10.10.0.0/16
cidr_numeral = 10
db_private_subnets = [
  "subnet-xxx",
  "subnet-yyy",
  "subnet-zzz",
]
env_suffix = d
private_subnets = [
  "subnet-aaa",
  "subnet-bbb",
  "subnet-ccc",
]
public_subnets = [
  "subnet-111",
  "subnet-222",
  "subnet-333",
]
region_namespace = apnortheast2
route53_internal_domain = dayone.internal.
route53_internal_zone_id = XYZ123...
shard_id = dayonedapne2
shard_short_id = dayone01d
vpc_id = vpc-xxxx
vpc_name = dayoned_apnortheast2

위의 예시처럼 output이 여러개 나오면 성공입니다. 콘솔에 가셔서 생성된 리소스를 한 번 확인해보시기 바랍니다.

Production 환경 구성

  • Production 환경은 위의 파일 세팅과 동일하게 진행하여 생성하시면 됩니다.

  • backend.tf 와 terraform.tfvars 에 있는 값만 Production에 맞게 변경해주시면 됩니다.

Prod용 폴더 생성

  • 앞에서 생성한 dayoned_apnortheast2 폴더를 통째로 복사합니다.

$ cp -r dayoned_apnortheast2 dayonep_apnortheast2
$ cd dayonep_apnortheast2
$ rm -rf .terraform

변수값 변경

backend.tf에서 저장될 버킷 이름을 변경합니다. 다만, 본 실습에서는 같은 계정으로 진행하므로 변경은 생략합니다.

vim terraform/vpc/dayonep_apnortheast2/backend.tf
terraform {
  required_version = "= 0.12.18"

  backend "s3" {
    bucket         = "dayone-prod-apnortheast2-tfstate"  << This should be changed!!!
    key            = "dayone/terraform/vpc/dayoned_apnortheast2/terraform.tfstate"
    region         = "ap-northeast-2"
    encrypt        = true
    dynamodb_table = "terraform-lock" 
  }
}

terraform.tfvars 에 실제 변수값을 변경합니다.

vim terraform/vpc/dayonep_apnortheast2/terraform.tfvars
aws_region = "ap-northeast-2"

# Production CIDR should be different from dev
cidr_numeral = "20"

# Please change "dayone" to what you want to use
# p after name indicates production. This means that dayonep_apnortheast2 VPC is for production environment VPC in Seoul Region.
vpc_name = "dayonep_apnortheast2"

# Billing tag in this VPC 
billing_tag = "prod"

# Availability Zone list
availability_zones = ["ap-northeast-2a","ap-northeast-2c","ap-northeast-2b"]

# In Seoul Region, some resources are not supported in ap-northeast-2b
availability_zones_without_b = ["ap-northeast-2a","ap-northeast-2c"]

# shard_id will be used later when creating other resources.
# With shard_id, you could distinguish which environment the resource belongs to 
shard_id = "dayonepapne2"
shard_short_id = "dayone01p"

# p means production
env_suffix = "p"

리소스 생성

Terraform plan / apply 를 통해서 리소스를 생성합니다. 과정은 위와 동일하므로 생략합니다.

만약, account가 다른 경우에는 해당 Account로 assume을 다시 하셔야 합니다.

PreviousVPCNextVPC Peering 생성

Last updated