Terraform on Colin Barker

Zero Trust Network Router on AWS

Fri, 12 Apr 2024 18:41:00 +0000

Header photo by Sander Weeteling on Unsplash

What is a “Zero Trust Network”

To put plainly, this is a network that is created that by default has Zero Trust within it, it is based off the idea of a Zero Trust security model which is a specific type of implementation used across different networks.

The main concept behind the zero trust security model is “never trust, always verify”, which means that users and devices should not be trusted by default
― Zero trust security model, Wikipedia

So, how does this apply to networking? Well the main concept here is, that you create a network that by default disallows traffic that isn’t known, verified, or wanted. This can be expanded, to then have all your devices connected into a single network, that by default, only allows traffic to pass between each node if you accept it. This is a concept that is often referred to as a “Zero Trust Network”.

Example of a simple network with switches and WiFi connection point

Starting with a basic scenario, above we have a simple office, it has a Data Room with a load of servers, with a couple of Virtual Machines, some Head office computers, a physical server, and a mobile phone connected to a WiFi network. Using this as a base, everything had to be in the same place, but it was all on the same network - devices would talk to each other through the switch, and traffic didn’t need to jump over anything other than the switch (excluding the phone!).

As the business expanded, it migrated the on-premise data room to a data centre, the Head Office was moved to a new location, and people had the option of working from home. Additionally, the CTO has asked, “We need to move to AWS”. Networking before this would have been quite complex to set up.

This is where a Zero Trust Network comes into play.

Depiction of a Zero Trust Network solution, with AWS hooked in

Let’s take a look at the above example. Even though the Head Office move was completed, the data centre had been set up, and there were workloads in AWS, we can see here - the network is still directly connected. With this scenario, each segment is part of the wider network, rather than using a VPN connection to route traffic between them, you can see that it is possible to just talk from one side to another, with the only hop being the Zero Trust network in the middle. Think of the provider as essentially a switch!

In this post, I will concentrate primarily on the Zero Trust Appliance in AWS, and how we can connect to a Zero Trust Network.

Introduction to Tailscale

Tailscale is one of the many different Software Defined, Zero Trust networking solutions that exist today. Many different providers have different ways of implementing their solutions, but they all are based on the same simple premise, “never trust, always verify”. Tailscale has multiple methods for adding devices to the network, by default, the Access Control Lists (ACLs) will not permit traffic between different devices unless explicitly states in the configuration of the ACL.

For anyone looking at behind the scenes of the technology used at Tailscale, I would recommend “How Tailscale Works” by Avery Pennarun who wrote how the data plan works, and a couple of examples as to why traditional VPNs might cause latency or even general issues in network connectivity.

Our example, Tailscale to connect an Office to AWS

Simply put, Tailscale will act as our “switch” and set up the point-to-point network between an EC2 instance, and an office where they might have several desktops.

For this post today, I will concentrate specifically on the Appliance that will sit in AWS, and how we would configure this for a customer.

Building a Zero Trust Network Router on AWS

For this, we will be using a specific set of tooling:

Terraform v1.7.5 - An infrastructure as code tool
AWS Provider - The translator between Terraform and the AWS API
Tailscale Provider - The translator between Terraform and the Tailscale API

Each “device” that is connected to Tailscale will need to be authenticated and approved before the device will be given access to the network. Even if there is an ACL in place that permits the access, the approval must occur. This can cause an issue when working with Infrastructure as Code (IaC), as the process would need to be automated. This can be overcome by generating a Tailnet Auth Key that is used specifically for the launching of the instance, and more specifically allowing your device to be added with “pre-approval”. For this, we will create a “tailnet key” resource.

# Create an authorization token for the Tailscale router to add itself to the Tailnet
resource "tailscale_tailnet_key" "tailnet_key" {
  preauthorized = true    # Set to true to allow the pre-approval of the device
  expiry        = 7776000 # Time in seconds
  description   = "Tailnet key for the server"
}

Simply put, this generates the key as a resource, and then Terraform knows what the key is, and any other properties that will have been shown by the API call to create the key.

With this key, the next step would be to install the Tailscale client onto the EC2 instance and make it into a router for any services within AWS. This would need to be done in a few stages when working with IaC, so we should start right at the beginning.

Tailscale offers a very comprehensive guide on how to install the tailscale client on a vast number of devices, including Linux, macOS, Windows, iOS, Android, and even down to Amazon Fire Devices, and Chromebooks too! What we are doing is creating a Subnet Router.

The Subnet Router is slightly different on a Zero Trust Network like this, while it is usually recommended to install the client on every single server, client, and virtual machine in the organisation, sometimes - it’s not needed. Even more so for organisations that use the Cloud, and have a LOT of ephemeral devices, and where the Software Defined Network of the Cloud Provider already has the security in place that keeps your network Well-Architected.

Within Terraform, there is a function called templatefile that can be used to generate strings or blocks of text that can use variables that are generated from within Terraform and then used within the string or block of text. Here, we are using the output of the tailscale_tailnet_key resource above, and pushing the generated key value into the user_data for an AWS EC2 Instance, to run a script on the first run. Below is the template used to install Tailscale.

#!/bin/bash

## Set the hostname of the server
hostnamectl hostname ${hostname}

## Ensure that the server is up to date with all the current packges
DEBIAN_FRONTEND=noninteractive sudo apt update -y
DEBIAN_FRONTEND=noninteractive sudo apt upgrade -y

## Enable IP Forwarding on the router to ensure that packets will flow as required
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

## Install Tailscale from the source
curl -fsSL https://tailscale.com/install.sh | sh

## Start Tailscale with the authorisation key to add it to the network
sudo tailscale up --authkey=${ts_authkey} --advertise-routes=${local_cidrs} --accept-routes

There is quite a bit happening in the script, it can be summarised as follows:

Set the hostname for the instance - While this is an EC2 instance, and usually in the cloud you would probably use more ephemeral devices, a Subnet Router will need to act more like a “pet” so that Tailscale sees this as an appliance object. Setting the hostname will mean that it is recognisable as to which host this is within your Tailscale network
Update the instance using apt - Pretty simple, make sure the instance is running the latest updates and patches before continuing!
Set IP Forwarding on the device - This is a key step, especially so in Linux, as this is to enable the EC2 instance to take traffic that it receives and forward it on. This specific setting tells the networking device at a kernel level to route traffic through it, as by default this is switched off.
Install Tailscale - The primary install of Tailscale, taken from the latest version on the Tailscale site.
Startup tailscale WITH the keys - Here we finally get to see where our tailscale key will be used - using the up function, we can set the authkey generated before, as well as which routes to advertise. We will go into this in a second, but for now, this is where the key will go.

Simple enough script, now we have to move on to creating the EC2 instance that will run

# Use a data object to get the latest version of Ubuntu
data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"]
}

# Build the Tailscale router using Ubuntu 22.04 LTS
resource "aws_instance" "this" {

  # General setup of the instance using the Ubuntu AMI
  ami                     = data.aws_ami.ubuntu.id
  ebs_optimized           = true
  instance_type           = "t4g.small"
  disable_api_termination = true

  # Key for SSH Access (if required)
  key_name = try(var.ssh_key_name, null)

  # Run the user_data for this instance to install Tailscale
  user_data = templatefile("${path.module}/templates/tailscale-install.sh.tpl",
    {
      ts_authkey  = tailscale_tailnet_key.this.key
      hostname    = var.hostname
      local_cidrs = var.local_cidrs
    }
  )

  # Networking Settings
  source_dest_check      = false # Disabled to allow IP forwarding from the network
  private_ip             = var.private_ip
  ipv6_addresses         = var.ipv6_addresses
  subnet_id              = var.subnet_id
  vpc_security_group_ids = [aws_security_group.this.id]

  .... (snipped)

}

A lot happening in this section, but in this example, we are using AWS Graviton (ARM-based) instances, as they are known to be a lot more efficient than other processor types, they can be cheaper, but also currently on AWS’s Free Tier for the time being!

The data object for the aws_ami - This will look for the latest version of the Ubuntu 22.04 image that exists in the Canonical account. Note the arm64 element of the filter string to look for the ARM version of the AMI.
The aws_instance resource to generate the subnet router - The standard for any EC2 instance, the primary resource with all its configuration
The general setup of the EC2 resource - Here we are using the data.aws_ami.ubuntu.id to ensure the right AMI is set, additionally making sure that api_termination has been configured, as we don’t want someone accidentally deleting the instance!
An SSH key - Some people might want to ensure they have an SSH key so they can log into the instance, in several cases this might not be needed.
User data template - This part is where we are taking the template bash file above, and taking the variables it knows about and entering details of what Terraform knows. Here we can see the ts_authkey variable is being set to the previously created tailscale_tailnet_key resource. Additional settings are also entered here.
Network Settings - This one contains one key variable that MUST be set for any EC2-based router that is created. The source_dest_check variable must be set to false. The source/destination checking is on by default, and within the software-defined VPC network on AWS, it will ensure that the traffic that sees, is the traffic for it - when it acts as a router, it will expect to see traffic pass through it from different devices that it needs to forward on. This check is disabled to allow this to happen. Without it, there is no way for traffic destined for another part of the network to flow to the EC2 instance.

Once this resource is launched, then it will appear in your Tailscale network, be approved, and should also be able to route traffic to and from your AWS network.

Some other elements that make up the design for this router, for example, you will need Security Groups set up to ensure traffic is allowed inbound and outbound to the EC2 instance and networks. To make this whole process easier, I created a Terraform Module that does all of this for you!

https://github.com/mystcb/terraform-aws-tailscale-router

In Part 2 of this blog post series, I will go into how to use this module to create a Multi-AZ version of this set up, and also include the changes I will be made to the module to enable Instance Recovery if the Operating System stops responding.

Examples of other Zero Trust Networking Solutions

- CloudFlare Zero Trust - This has more recently had an update that will allow access through one of its WARP clients. This is still in beta so one to keep an eye on. - Enclave - This was my first ever experience with Zero Trust networking, so I can’t not name-drop this one! While personally, I don’t use it anymore, it was here I learnt the basics of the Zero Trust network before moving myself to Tailscale.

Ten Reasons to make Game Days a regular event

Tue, 24 Oct 2023 22:18:28 +0000

Header photo by Marvin Meyer on Unsplash

What is a game day?

A game day sees a group of people solving a fictional problem or developing a new idea. They’re given a short period of time – usually hours, but sometimes days or a couple of weeks – to complete the challenge. The beauty of game days is that they’re inherently flexible. Participants can meet in person or virtually. They might be existing team members and colleagues, or complete strangers. They could have shared skills and experiences or come from a wide range of backgrounds and disciplines.

Some of the best events I’ve been part of involve technical and non-technical people working together. Having both organised and participated in game days, I think they’re an excellent way to foster a growth mindset. They can also play a valuable role in change management if you’re introducing new ways of working or integrating teams. Game days are great fun, but they also deliver serious business benefits.
Ten great things about game days

Strengthen relationships

The time constraints of game days create a high-octane environment. As pressure mounts to complete the task participants have to rely on each other to get jobs done. This can result in a feeling of camaraderie that lasts far longer than the event itself. It’s a great way to build bridges and nurture relationships between departments, global teams or people with different levels of experience.

Extend skills

Game days often require participants to draw on a wide range of skills. The core challenge might be technology focused, but solving it is likely to require leadership, mentoring, communication and decision making. Participants are often pushed out of their comfort zone, and it’s great to see people rise to the occasion. You may discover colleagues have hidden talents that make them a great fit for future projects.

Push boundaries

We all know that it’s important to drive continual improvement, but it’s also easy to get stuck in the rut of day-to-day tasks. Game days encourage you to think outside the box. They offer a safe space to push boundaries and deal with any consequences. Sometimes this reveals new and better ways of working that can be implemented in the real-world.

Try new things

One of the best – and most stressful – games days I took part in saw the organiser continually introducing faults that we had to go and fix. We’d complete one task, and another bigger problem would emerge. As the challenges escalated, I found myself reaching for tools and techniques that I’d read about but hadn’t yet used. It was a great opportunity to try new ideas and learn on the fly.

Road-test plans

Game days can be a powerful test bed for activities like cloud disaster recovery. Having a plan in place is all well and good, but staging a worst-case scenario enables any gaps to be identified and rectified. This can form a central part of regular disaster recovery reviews.

Plan for failure

Look at your architecture and think about where things could go wrong or what a malicious actor might do. Structuring a game day script around this can be an interesting way to test the resilience of cloud-based systems.

Encourage innovation

The open-ended nature of game days, coupled with their removal from day-to-day priorities, makes them a fertile ground for innovation. They can also create a dynamic space for people with different perspectives and capabilities to spar and stimulate each other. While the timeframe is limited, teams can be surprisingly productive. One game day I hosted resulted in a team developing a chatbot to solve a business challenge. Soon afterwards it was launched on the company’s website.

Solve problems or avoid future issues

Building a game day around a specific business goal or priority can be very effective. In a cloud context, you could take a theme like security or cost-efficiency and use that as the focal point. The event might deliver tangible outcomes or ideas that can be implemented in the real world. At the very least, it will give participants a deeper awareness and understanding of the topic.

Foster psychological safety

Cloud adoption can be really hard on individuals as they get to grips with new ways of working. People tend to adapt more quickly and embrace the inevitable challenges more willingly if their team and the wider workplace is psychologically safe. This is a huge cultural issue, and game days can’t be used as a sticking plaster solution. But they can help building trust and creating an environment where people are comfortable with experimentation and don’t fear failure.

Accelerate progress

Game days are a brilliant vehicle to improve cohesion between and within teams. They encourage people to stretch their capabilities, think differently and try new things. They’re exciting, stimulating and challenging. All of this brings out the best in people, sharpening their skills and refreshing their perspectives. And it greases the wheels to enable quicker, more seamless progress with game-changing initiatives like largescale cloud adoption.

Enabling IPv6 on AWS using Terraform - AWS Network Firewall (Part 3)

Wed, 12 Jul 2023 19:40:58 +0000

Header photo by NASA on Unsplash

⚠️ Note: This is Part 3 of my IPv6 on AWS series, the first part is available here, and the second part here). ⚠️

Introduction

With this post, we look into how an AWS Network Firewall can be used in a DUALSTACK mode to cover both IPv4 and IPv6. In a future post, we will go into more depth into how we can enable this for IPv6 only, but in this case we are using this as a stepping stone. The main reason for me to go into this detail is finding this throughout the current eco system seemed to be missing all the key steps in a single location! I am sure there will be soon, but ultimately this was my experience and how we can get over to an IPv6 world.

Where to begin?

To start, I will be taking over from one of the AWS Network Firewall example architectures. A AWS Network Firewall with a NAT Gateway. Instead of going over essentially what is already a known design pattern, I will just cover enough to get us started.

Standard IPv4 AWS Network Firewall Design

Diagram a standard IPv4 network, using AWS Network Firewall and a NAT Gateway

The AWS Network Firewall is a centralised managed Firewall Appliance that allows you to scale and protect your workloads in AWS. It uses the AWS Gateway Load Balancer and GENEVE Protocol the that we have covered in a previous post What is an AWS Gateway Load Balancer anyway?. The main difference is that the EC2 appliance we used in this post, is replaced with the AWS managed service.

Where does IPv6 fit in?

Looking back at part one, I mention in Adding IPv6 outbound routing to the private subnets that with IPv6 networks, having a NAT to expand the IP ranges, isn’t really needed. You can assign each compute element with its own publicly routable address.

So with this in mind, the “application EC2 instance” seen in the design above, would get it’s own IPv6 address, and wouldn’t need to be NATted.

Terraform behind the IPv4 solution

Let’s start with the original diagram. The code for this is up at 04-network-firewall-ipv4 if you wish to see the whole repo, but here are snippits of the important bits.

Something I have had a few issues with, was getting a easier mapping from the AWS Network Firewall to the VPC Endpoints in each subnet, so within the locals.tf file, I have generated this block to export the mappings. It sets the key for each of the endpoints as the availability zone that it has been configured in, which will be handy later on when we try and map to the right routing table.

# Generate a list of the network firewall endpoints so the route tables can use them
locals {
  networkfirewall_endpoints = { for i in aws_networkfirewall_firewall.firewall.firewall_status[0].sync_states : i.availability_zone => i.attachment[0].endpoint_id }
}

Next we have the routing table for the Internet Gateway. As mentioned my previous post under How does this work in AWS, the key to routing traffic inbound to the right location, is an Edge Associated VPC Route table. In summary, these are added to the edge, and the Internet Gateway (IGW) to override the standard routing, and push traffic directly to the Gateway Load Balancer (GWLB) Endpoint for the Network Firewall.

# Starting with the route table to be assigned to the Edge, this is the Internet Gateway's route table
resource "aws_route_table" "igw" {
  vpc_id = aws_vpc.example.id
}

resource "aws_route" "igw_to_firewall" {
  # We use a count here to go over each of the NAT subnets that exist, to create a route for each based on the Availability Zone
  count = length(var.nat_subnets)

  route_table_id         = aws_route_table.igw.id
  destination_cidr_block = var.nat_subnets[count.index]
  vpc_endpoint_id        = local.networkfirewall_endpoints[var.availability_zones[count.index]]
}

# Associate the Route table to the Edge IGW
resource "aws_route_table_association" "igw" {
  gateway_id     = aws_internet_gateway.transit.id
  route_table_id = aws_route_table.igw.id
}

For the NAT Gateway Subnet, we would have the return route back into the Network Firewall for outbound traffic.

# Creation of each of the NAT Gateway subnet route tables that point to the Network Firewall
resource "aws_route_table" "nat" {
  # We are using a count here, because we need to create a route table for each Availability Zone
  count = length(var.nat_subnets)

  vpc_id = aws_vpc.transit.id
}

# Create a route that tells the NAT network to route traffic to the internet via the NWF
resource "aws_route" "nat_to_firewall" {
  # We are using a count here, because we need to create a route for each Availability Zone
  count = length(var.nat_subnets)

  route_table_id         = aws_route_table.nat[count.index].id
  destination_cidr_block = "0.0.0.0/0"
  vpc_endpoint_id        = local.networkfirewall_endpoints[var.availability_zones[count.index]]
}

# Associate the Route table to the NAT Subnets
resource "aws_route_table_association" "nat" {
  # We are using a count here, because we need to create an association for each Availability Zone
  count = length(var.nat_subnets)

  subnet_id      = aws_subnet.nat[count.index].id
  route_table_id = aws_route_table.nat[count.index].id
}

And then for the Private Subnets where the applications, EC2 instances, or any service that requires access to the internet from a Private Subnet, we will need to route all traffic through to the NAT Gateway in each of the Availability Zones.

# Creation of each of the private route tables that point to the NAT Gateway
resource "aws_route_table" "private" {
   # We are using a count here, because we need to create a route for each Availability Zone
  count = length(var.private_subnets)

  vpc_id = aws_vpc.transit.id
}

# Create a route that tells the private network to route traffic to the NAT GW in each AZ
resource "aws_route" "private_to_natgw" {
  count = length(var.private_subnets)

  route_table_id         = aws_route_table.private[count.index].id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.nat[count.index].id
}

# Associate the Route table to the Private Subnets
resource "aws_route_table_association" "private" {
  count = length(var.private_subnets)

  subnet_id      = aws_subnet.private[count.index].id
  route_table_id = aws_route_table.private[count.index].id
}

So far, pretty much standard for a GWLB setup. This should cover everything that is needed to get the routing element working for the VPC.

Lets bring in IPv6

Solution Design

Updated Diagram showing the IPv6 routes in a DUALSTACK setup

With some minor adjustments to our routes, we are able to add the correct routing in place for IPv6 to pass traffic through the AWS Network Firewall. The difference being, is that anywhere that can have an IPv6 address, we would route it directly to that Availability Zone’s Network Firewall Endpoint (The Gateway Load Balancer endpoint).

Some major changes though, will be having to convert the existing IPv4 AWS Network Firewall endpoints into what is known as a DUALSTACK address type. This however, isn’t as easy as just updating the Terraform. As per the documentation it is not possible to change the IPAddressType after you have set the subnet.

However, if you were to just change the value in Terraform, you will receive the following error message from Terraform:

Error: associating NetworkFirewall Firewall (arn:aws:network-firewall:eu-west-2:012345678910:firewall/vpc-network-nfw) subnets: InvalidRequestException: subnet mapping(s) is invalid. You can't change the IP address type of an existing subnet. Either remove the subnet from the request or change the IP address type to match the subnet's original value, and try again, parameter: [[{"subnetId":"subnet-01234567891012345","ipaddressType":"DUALSTACK"},{"subnetId":"subnet-5432109876543210","ipaddressType":"DUALSTACK"},{"subnetId":"subnet-1111111111111","ipaddressType":"DUALSTACK"}]]

In this example, we have three subnets as part of our solution, of which all three were originally setup as “IPV4”. So we are given two options for migration.

Remove the AWS Network Firewall and redeploy with the “DUALSTACK” setting on each subnet
Manually change the firewall settings, and re-import back into the Terraform state.

In a lot of cases, the first option will be hard because you can’t always delete the Firewall that might be in production, and the second option you have the issue that you can’t add in a new endpoint in the same availability zone while the previous one exists. However, making the manual changes is never in the spirit of IaC.

That being said, the steps to do a manual change would be with minimal outages, but also potentially additional cross AZ networking fees would be to do the following:

Complete once for each availability zone

Replace the route on the Edge Associated route table that points to the Gateway Load Balancer endpoint for that Availability Zone to point at another subnets endpoint
Replace the route from the Route Tables associated to the NAT Gateway subnets that point to the Gateway Load Balancer for that Availability Zone to point at another subnets endpoint
Edit the AWS Network Firewall and remove the endpoint from the list of Firewall Subnets for that Availability Zone only - save the changes and wait
- ⚠️ NOTE While the console will show that it has successfully updated, it will error out if it is still deleting the endpoint, this could take up to 20 minutes to complete. ⚠️
Re-edit the AWS Network Firewall and add back in the subnet specifically selecting “DUALSTACK” for the IP Address Type, and hit save.
Replace the route on the Edge Associated route table that points to the other subnet, back to the new Gateway Load Balancer endpoint for original subnet.
Modify the Terraform to switch the ip_address_type in the subnet_mapping block to be DUALSTACK

At this point, run a Terraform Plan to ensure that the state and the Terraform code match up.

⚠️ NOTE This process can take a few hours to complete! ⚠️

Terraform behind the IPv6 solution

Moving on from the above, at this point we need to add in the routing so that the subnets can use their IPv6 addresses to connect to through the AWS Network Firewall. If you would like to see the code directly, there is a second version of the code up at 05-network-firewall-dualstack.

IPv4 and IPv6 Routing

A rule of the VPC Route Tables, is that you need to separate out your routes for IPv4 and IPv6. This can come in handy later on when we discuss what to do in place of the NAT Gateway, however, this example below shows you what a route table should look like for a public facing subnet behind the AWS Network Firewall.

Routing on the Edge

The route table here will be a little simpler, as all traffic needs to head to the Gateway Load Balancer VPC endpoint for the Availability zone that it is in. In this case, this can be generated from knowing which of the IPv6 networks are in each Availability Zone.

A route table showing both an IPv4 and IPv6 route to the Gateway Load Balancer VPC Endpoint and the NAT Gateway on the Edge Associated route

Routing in the private subnet

A route table showing both an IPv4 and IPv6 route to the Gateway Load Balancer VPC Endpoint

In this specific case, we can see that both the IPv4 “Route All” 0.0.0.0/0 route goes to the same VPC Endpoint as the IPv6 “Route All” ::/0 route. This route table is generally used for the Public Subnet behind an AWS Firewall, or in our case the NAT Gateway Subnet. Here the target of both is the same endpoint set up for this Availability Zone.

Routing in the NAT subnet

One major difference comes in specifically on the Private or Application Subnets. These are the ones that typically you would have routed through your NAT Gateway, which in an IPv6 world, you wouldn’t need to do, as every device can have its own IPv6 address.

⚠️ NOTE In a future post, I will go into methods to ensure that IPv6 addresses are hidden, however, for the use case of the AWS Network Firewall, it isn’t currently possible to do this. ⚠️

A route table showing both an IPv4 and IPv6 route to the Gateway Load Balancer VPC Endpoint and the NAT Gateway for a private network

As you can see in the above image, we are routing IPv4 traffic as normal to the NAT Gateway for the Subnet, but for the IPv6 traffic, we are routing that to the same Gateway Load Balancer VPC Endpoint that we had in this Availability Zone. As services in the IPv6 subnet will have their own publicly routable IPv6 address, this means that we can bypass the NAT Gateway.

Terraform for routing IPv6

For the Edge Route table, we will be sending the new IPv6 traffic destined for the Private Subnet, to the Gateway Load Balancer VPC Endpoint in each of the availability zones.

# Create a route that sends all IPv6 traffic for the Private Subnets to the NWF Endpoint
# This is due to IPv6 not being NAT'd, so each application gets its own IPv6 address
resource "aws_route" "igw_to_firewall_ipv6" {
  count = length(var.private_subnets)

  route_table_id              = aws_route_table.igw.id
  destination_ipv6_cidr_block = aws_subnet.private[count.index].ipv6_cidr_block
  vpc_endpoint_id             = local.networkfirewall_endpoints[var.availability_zones[count.index]]

}

For the NAT Subnet and the Private Subnet, both of these routes will end up being the same, as technically we have made the Private Network routable through the use of IPv6 addresses, as such we can include the following two bits of code

# Create a route that tells the private network to route IPv6 traffic to the NAT GW in each AZ
resource "aws_route" "private_to_natgw_ipv6" {
  count = length(var.private_subnets)

  route_table_id              = aws_route_table.private[count.index].id
  destination_ipv6_cidr_block = "::/0"
  vpc_endpoint_id             = local.networkfirewall_endpoints[var.availability_zones[count.index]]

}

# Create a route that tells the NAT network to route IPv6 traffic to the internet via the NWF
resource "aws_route" "public_to_firewall_ipv6" {
  count = length(var.nat_subnets)

  route_table_id              = aws_route_table.nat[count.index].id
  destination_ipv6_cidr_block = "::/0"
  vpc_endpoint_id             = local.networkfirewall_endpoints[var.availability_zones[count.index]]

}

Within Terraform, we can add the routes for each of the route tables as follows.

In Summary

AWS Network Firewall is a great product when used in the right way, and ensuring the right routing is in place to make this work is a major key point in this. Hopefully, my own pathway to make this work has helped you out, as this took me a long time to actually get working in the end!

There are still some elements that I believe do need improving on, for example - I have made a typically private network into a public network, albeit behind the firewall. While not having a NAT Gateway is one of the key benefits of IPv6 networking, it would still be an issue for some customers who really did need a secure private network. While there is the option for the Egress Only Outbound Gateway, currently this won’t sit behind the AWS Network Firewall, and ultimately provides a route to the internet that isn’t filtered in anyway. One suggestion would be to use a NAT instance, or other device to route traffic through, but this will be a post for another day.

One final element to this journey, just a few months ago AWS announces IPv6 only networking support for the AWS Network Firewall. We shall use this as part of the future post on an IPv6 only VPC.

Thanks again for reading!

Enabling IPv6 on AWS using Terraform - EC2 "Pet" Instance (Part 2)

Sat, 04 Mar 2023 14:26:18 +0000

Header photo by NASA on Unsplash

⚠️ Note: This is Part 2 of my IPv6 on AWS series, the first part is available here. ⚠️

What is a “pet” instance, and why?

Before I start, this is an exceptional use case. When using AWS you should always use ephemeral instances, it is always best practice. There are times however, you might need to do this to a singular instance for any number of very valid reasons. In my case, I have a very cheap proxy service between the outside world any my home network. I use this external service as a very cheap method to enable access to some systems that can be accessed through a Site-to-Site VPN to my home. You might have other reasons too, for example - Microsoft Active Directory servers which are hosted on EC2 instances would be considered “pet”.

In my case, I have this one static instance using an Elastic IPv4 IP, and I would like to give it an IPv6 address.

Pets vs Cattle Analogy

cattle, not pets
― Bill Baker, Twitter

This phase used during a presentation about Scaling SQL Servers way back in 2006, to show how attitudes towards computing evolved since the early days.

It describes the idea that servers can be two types:

Pets: These are your pride and joys, there is just one Oz the Cat that sits on your lap while writing blog posts about IPv6, you look after them, nurture them, and you deal with everything that comes their way as and when it happens.
Cattle: You have a farm, you have a vast amount of animals that help you produce several products that you sell to the market. If one of those animals becomes an issue, you “replace” them. No sentimental attachment.

I will say, this analogy will probably not last the test of time - the latter “Cattle” explanation can upset a few people for different reasons and you probably will see this change in the future. Hoping for something like “Cooker vs Food” or “House vs Tent”, but I can’t see that happening any time soon!

So with that in mind, you can see the following:

Pet Instances: They have a set hostname, they have had loads of love, care, and attention given to them. When they go wrong, you investigate, identify the issues, remediate, and bring back to health. They also make lots of noises when you need to give them attention. (Yes, Oz is meowing at me at the moment!)
Cattle Instances: They have a randomly generated unique identifier, you keep the safe and secure, but once the instance starts to fail, you quickly take them out of the loop, and replace with a healthy instance. The failed instance, you just get rid of.

Starting Point

Before we begin, we need to set the scene a little. What we will be working with is incredibly simple - an EC2 instance sitting in a Public Subnet.

Very basic setup of an EC2 instance in a public subnet

You can find the code to deploy the above diagram on my GitHub - IPv6 on AWS repo. Feel free to follow along in your own sandbox account if you wish!

The main block of code we need to start looking at is the ec2_instance resource, there are some comments inline to explain what we are doing.

# Build the EC2 instance using Amazon Linux 2
resource "aws_instance" "test" {
  ami                     = data.aws_ami.amazon_linux_2.id   # Dynamically chosen Amazon Linux AMI
  ebs_optimized           = true                             # EBS Optimised instance
  instance_type           = "t4g.nano"                       # Using a Graviton based instance here
  disable_api_termination = true                             # Always good practice to stop pet instances being terminated

  # Networking settings, setting the private IP to the 10th IP in the subnet, and attaching to the right SG and Subnets
  source_dest_check      = false
  private_ip             = cidrhost(aws_subnet.public_a.cidr_block, 10)
  subnet_id              = aws_subnet.public_a.id
  vpc_security_group_ids = [aws_security_group.test_sg.id]

  # This requires that the metadata endpoint on the instance uses the new IMDSv2 secure endpoint
  metadata_options {
    http_endpoint = "enabled"
    http_tokens   = "required"
  }

  # Sets the size of the EBS root volume attached to the instance
  root_block_device {
    volume_size           = "8"      # In GB
    volume_type           = "gp3"    # Volume Type
    encrypted             = true     # Always best practice to encrypt
    delete_on_termination = true     # Make sure that the volume is deleted on termination
  }

  # Name of the instance, for the console
  tags = {
    "Name" = "Sample-EC2-Instance"
  }

  # Ensures the Internet Gateway has been setup before deploying the instance
  depends_on = [
    aws_internet_gateway.sample_igw
  ]
}

Nice and simple really, but this is where simple can cause an issue in this case.

The Issue

Without going to too much detail, as you know Terraform uses a State that will store details about the environment it manages, including the settings and configuration of resources. It will use this information to keep track of what it knows, and what has changed - giving it the great ability to see what will change during the next application of your code. This state is generated from the outputs of the providers API’s and your code.

However, the API call for creating an EC2 instance does more than just create a single EC2 resource. This also happens with multiple services, and multiple cloud providers as well, so it isn’t specific to AWS on this case. The single API call makes the process a lot simpler to build up the compute for you, but it includes one additional resource: The Elastic Network Interface.

In typical use, this is great, it saves you building instances with no networking and then having to mess about with it later on. I would also point out, that a cloud service with no networking access, is no different to getting a physical computer, disconnecting everything except the power, wrapping it in concrete, and then seeing what you can do. I mean it will still run (albeit very hot and probably melt), but you can’t then do anything with it, or see what is happening!

EC2 instance showing the Elastic Network Interface (ENI) attached

Terraform does have a resource for the Elastic Network Interface (ENI) however, as the API created this resource itself, attached it to the instance using the parameters specified in the ec2_instance resource, Terraform doesn’t know about this. Herein lies the issue.

Adding an IPv6 Address using the CLI

Let’s say you don’t have your Infrastructure defined as code and you needed to assign an IPv6 address to your instance, thankfully the awscli has such a method. Under the ec2 service type, there is a command to assign-ipv6-addresses (Documentation)

To do this, you would run the following command:

aws ec2 assign-ipv6-addresses --ipv6-addresses <address> --network-interface-id eni-007ed4d597d6df6b7

The one item you need to know is, the network-interface-id. While the ec2_instance resource, does have this as an output, the only way to get this is after the resource has been created.

What happens in Terraform?

When using the AWS EC2 API to create the instance, behind the scenes it will create that interface using the settings, and return the Interface ID. As the API’s are not supposed to keep the State themselves, it will always consider a change to those settings in the creation block, as a new interface/network settings.

As we define the network settings in our resource for the EC2 instance, Terraform knows there is a change to the state, talks to the AWS EC2 API, that then will require the network interface to be recreated. The only way to do that would be create a new interface, detach the old interface, attach the new interface, and you are done, except, that isn’t possible. AWS Documentation on Network Interfaces state that “Each instance has a default network interface, called the primary network interface. You cannot detach a primary network interface from an instance.” This comes into play with my workaround in this blog later on. Therefore, the only option, is to re-create the instance.

If we were to simply use the ipv6_addresses in the ec2_instance block, to add a new IPv6 address, Terraform will report the following:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

# aws_instance.test must be replaced
-/+ resource "aws_instance" "test" {

      <<SNIP>>

      ~ instance_initiated_shutdown_behavior = "stop" -> (known after apply)
      ~ instance_state                       = "running" -> (known after apply)
      ~ ipv6_address_count                   = 0 -> (known after apply)
      ~ ipv6_addresses                       = [ # forces replacement
          + "2a05:d01c:b90:ee00::10",
        ]
      + key_name                             = (known after apply)
      ~ monitoring                           = false -> (known after apply)
      + outpost_arn                          = (known after apply)

      <<SNIP>>

    }

As you can see, adding the address forces replacement. Which for our wonderful pet instance here, is a terrifying thought!

Workaround - Pseudo Code

⚠️ Note: This is not going to be the best way, in all honesty this is a hack more than a workaround, but it does make it easier to bypass the issue. Always consider backups before doing this, always consider that if you need to do this, why do you need to do it? ⚠️

Given that the AWS CLI can add an IPv6 address without rebuilding the instance, as well as the console as well, it does mean that a replacement isn’t needed to add the address on. For this, we have to play around a bit with Terraform, run it a few times, to get everything up in sync. To this we will need to:

Create an aws_network_interface resource to match the primary ENI with the IPv6 address
Import the ENI resource into the Terraform State (requires CLI)
Apply the changes using Terraform, and confirm the IPv6 address has been attached
Add the ipv6_addresses list to the ec2_instance resource block
Delete the aws_network_interface from the Terraform State (requires CLI)
Delete the aws_network_interface resource block from the code
Run a plan to confirm everything is working

As you can see, there are a few steps to do this, but it does mean that if you ever need to re-deploy the code, this will create an identical copy of the instance (minus data) and matches the code.

This also removes the issue later on if you ever wish to destroy the code. If you were to stop after the Apply when you have the IPv6 address assigned, you will now have an aws_network_interface resource block managed by Terraform, but also managed by the EC2 Instance itself. As mentioned before, you cant remove the primary network interface from the EC2 instance, and if you try and run a destroy, Terraform will attempt to call the API to tell AWS to delete the interface, an come back with a 400 error.

Error: detaching EC2 Network Interface (eni-071f55452ccc18997/eni-attach-0a424f74a43a7a0af): OperationNotPermitted: The network interface at device index 0 cannot be detached.

So the workaround requires us to complete all the steps to make this work.

Workaround - Terraform

Create the ENI

First we need to create the aws_network_interface resource block. We will need to try and match as much as we can to what we have defined in the ec2_instance block. Below is an example of of this block created using the sample code above.

resource "aws_network_interface" "test_eni" {
  subnet_id         = aws_subnet.public_a.id
  # Uses the same IP from the ec2_instance resource
  private_ips       = [cidrhost(aws_subnet.public_a.cidr_block, 10)]
  # The new IPv6 to assign - note that 16 in hexadecimal is 10
  ipv6_addresses    = [cidrhost(aws_subnet.public_a.ipv6_cidr_block, 16)]
  # Same security group from the ec2_instance resource
  security_groups   = [aws_security_group.test_sg.id]
  # Continue with additional settings from the ec2_instance resource
  source_dest_check = false

  # Match the attachment details on the EC2 instance
  attachment {
    instance     = aws_instance.test.id
    device_index = 0
  }

}

Import the ENI into State

Next we need to get the ENI information into the state, to do this we need the network interface ID from AWS. This can be done through the console, or by looking at the State file (if you can). In our example, the ENI is eni-071f55452ccc18997. Terraform at the bottom of all of its documentation will have the command you need to import the resource in, in our case we will need to import this ENI into this resource block

terraform import aws_network_interface.test_eni eni-071f55452ccc18997

Apply the changes

As we have already created the block with the IPv6 address in it, we should be able to run the plan and apply to add the IPv6 address. Do remember to check your plan! Make sure that it is doing what you expect, in my case the plan showed:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_network_interface.test_eni will be updated in-place
  ~ resource "aws_network_interface" "test_eni" {
        id                        = "eni-071f55452ccc18997"
      ~ ipv6_address_count        = 0 -> (known after apply)
      ~ ipv6_address_list         = [] -> (known after apply)
      + ipv6_address_list_enabled = false
      ~ ipv6_addresses            = [
          + "2a05:d01c:b90:ee00::10",
        ]
      + private_ip_list_enabled   = false
        tags                      = {}
      ~ tags_all                  = {
          + "Environment" = "Sandbox"
          + "Source"      = "Terraform"
        }
        # (15 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

As shown, the only major change will be that there is an additional IPv6 address assigned to it, but also my default tags are added too.

The EC2 instance now has an IPv6 address associated with it

Technically at this point, we have done it - but we still have to sort out our Terraform to prevent future issues.

Add the IPv6 address to the instance block

Thankfully the block we created, also has the same line that we can copy back into the aws_instance block, we can sneak this back in to match the state in AWS>

# Build the EC2 instance using Amazon Linux 2
resource "aws_instance" "test" {

  <<SNIP>>

  # Networking settings, setting the private IP to the 10th IP in the subnet, and attaching to the right SG and Subnets
  source_dest_check      = false
  private_ip             = cidrhost(aws_subnet.public_a.cidr_block, 10)
  # IPv6 Address for the instance from the aws_network_interface block
  ipv6_addresses         = [cidrhost(aws_subnet.public_a.ipv6_cidr_block, 16)]
  subnet_id              = aws_subnet.public_a.id
  vpc_security_group_ids = [aws_security_group.test_sg.id]

  <<SNIP>>
}

Delete the ENI from the State

Before we remove the block, we need to remove the information from the State. This will prevent us from accidentally deleting the block, running an apply, and getting the 400 error mentioned before! To do this we will need to run the following command:

terraform state rm aws_network_interface.test_eni

Now you can delete the whole aws_network_interface block from your code.

Run a plan to check

With all this complete, we just need to run one very last terraform plan to confirm, if everything has gone right then your output should be:

1
2
3

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Summary

Just would like to point out, this is not the ideal way of doing this. I am sure there are other ways, but this is how I got around the issue. Thankfully in my case I had access to the Terraform State, which made the addition of the aws_network_interface a lot easier. This workaround doesn’t work too well when you do not have access.

This technique can work not just on AWS, but other cloud providers too - its mainly about the logical steps of importing the unmanaged resource, making the changes, and then releasing it back to AWS to manage.

However, we are reminded here why pet instances are just that, sometimes they can be a bit of a pain, but you nurture them for a reason! In my case, I am just a little bit lazy, and didn’t want to have to set everything up again!

Hopefully I will continue this IPv6 series soon, where I will go over a number of other services - to see if we can’t push forward with the IPv6 transition.

Any comments or queries will be greatly appreciated!

Oz The Cat

For anyone that is wondering, this is Oz, he is a lovely cat!

This is Oz!

Enabling IPv6 on AWS using Terraform (Part 1)

Sat, 11 Feb 2023 15:08:20 +0000

Header photo by NASA on Unsplash

What is IPv6?

According to Wikipedia, Internet Protocol version 6 (IPv6) was introduced in December 1995 (just over 25 years ago!), based on the original IPv4 protocol that we all know and love today. The Internet Engineering Task Force (IETF) developed this new protocol to help deal with the (at the time) anticipated exhaustion of the IPv4 address range. This process seemed like it could be simple enough, create a new system to replace and older system and enable the expansion of the Internet to meet modern day standards.

This is where the problem lie. Adopting IPv6 wasn’t as simple as just replacing IPv4. The two protocols are different enough that on top of physical hardware changes (older devices unable to support IPv6), it also meant a different way of thinking when it came to working with networking both on the Internet, and within (Intranets, local networks, home networks). One of the biggest issues that faced a lot of the world, is that ISP adoption rate was surprisingly low.

However, as the IPv4 Exhaustion happened as early as 2011, providers have started very quickly adopting the new standard.

So, what is IPv6?

I would recommend reading the Wikipedia for more details, as much of what I would write here, would essentially be copied just from that page! In summary though"

IPv6 uses 128-bit addresses over the IPv4 32-bit addresses, allowing approximately 3.4 x 10^38 addresses, over IPv4’s 4,294,967,296 (2 x 10^32)
Addresses are in 8 groups of hexadecimal digits, separated by colons. For example: fd42:cb::123 in short hand. (which would expand to be fd42:00cb:0000:0000:0000:0000:0000:0123)
Route Aggregation is built into the addressing scheme allowing for the expansion of global route tables with minimal space used.
Steve Leibson (in a now lost article) one said “[If] we could assign an IPv6 address to every atom on the surface of the earth, [we would] still have enough addresses left to do another 100+ Earths!”

⚠️ Note: I mentioned short hand above, and this comes with a lot of caveats but the primary rule is, you can drop any zero in an IPv6 address, and it is assumed, as long as it is before the hexadecimal character. For example 00cb:0000:0001 can be shortened to cb::1. However, you can only have ONE :: in each address (otherwise how does it know where to shorten it!), so for example 00cb:0000:0100:0000:0001 must be shortened to cb:0:100::1. I’ll go into this in more detail in a future post!⚠️

This is why it is important to start embracing IPv6, we have a lot more space to make lives easier for the world, and the only way we can ensure the continued adoption of the protocol is to enable this everywhere.

In this post, I will go through how you can enable, using Terraform, IPv6 on your existing AWS Cloud Networks. When planning IPv6, there is a lot to consider, and there are some architectural changes will need to be considered.

Setting the scene - an existing AWS Cloud Network

Basic AWS VPC with Networking

This should be a very familiar layout to most people, an VPC in AWS with some very basic networking setup. In the diagram above, we have a VPC, with Public and Private Subnets in two availability zones. We have an Internet Gateway for the public subnets, and a NAT Gateway in each Availability Zone for the Private Subnets to talk to the internet. I have placed some sample IP addressing in, just for reference as part of this post.

If you wish to deploy this yourself, I have place some sample code on my GitHub that you can use. (https://github.com/mystcb/ipv6-on-aws/tree/main/01-sample-vpc)

Throughout this post, you will see me mention the cost of running this using an estimate. I have been using for a while, a tool called infracost which is an open source (with subscription based additions) cost estimator tool - https://www.infracost.io/. For this demonstration, using the sample code listed above, it would cost an estimated $76.65/month - so if you don’t want rack up a bill, only deploy when you want to test, and use Terraform to destroy the services when you are done.

As an example, here it the cost estimate for this deployment:

# infracost breakdown --path=.

Evaluating Terraform directory at .
  ✔ Downloading Terraform modules
  ✔ Evaluating Terraform directory
  ✔ Retrieving cloud prices to calculate costs

Project: mystcb/ipv6-on-aws/01-sample-vpc

 Name                                     Monthly Qty  Unit            Monthly Cost

 aws_eip.natgwip[1]
 └─ IP address (if unused)                        730  hours                  $3.65

 aws_nat_gateway.sample_natgw_subnet_a
 ├─ NAT gateway                                   730  hours                 $36.50
 └─ Data processed                      Monthly cost depends on usage: $0.05 per GB

 aws_nat_gateway.sample_natgw_subnet_b
 ├─ NAT gateway                                   730  hours                 $36.50
 └─ Data processed                      Monthly cost depends on usage: $0.05 per GB

 OVERALL TOTAL                                                               $76.65
──────────────────────────────────

Remember, all costs are estimates! With the cloud, its pay as you use, utility based so the costs will be what you use. The above is just an estimate, so keep that in mind!

IPv6 and AWS

As mentioned before, there are some concepts that have to be considered when designing for IPv6. One specific concept for networking can seem a little confusing at first, but with the right security in place, will ensure that no accidental access to the service can happen.

Within AWS, IPv6 addresses are global unicast addresses. A unicast address is an address that identifies a unique object or node on a network. While the premise of a unicast address is not new (as it was the same in IPv6), with the exhaustion of IPv4 addresses, new methods to give unique addresses to multiple nodes was devised. Network Address Translation (NAT) was one such method for doing this. This allowed the mapping of a single unicast IP address to be used by multiple nodes, by routing traffic through the NAT node and allowing it to re-write the headers to make it seem like it had come from the unique address.

NAT is a wide subject, and I am sure I will write more about it some day, but a real world example that you see in most places is your home network. You have multiple private nodes with access to the internet, typically using a single public unicast address.

So how does this relate to IPv6 and AWS? Remember earlier, I mentioned that there are so many addresses available in the IPv6 ecosystem that you could give a unique address to every atom on the planet? Well, we can do just that, but to the nodes that require addresses. This is what AWS does, each IPv6 address you assign to nodes in AWS, are global unicast addresses, unique to each node.

This means a “private” IPv6 Subnet does sound like it might be complicated to set up, but actually it isn’t as bad as you think! However, to start the process off, we will start with adding a IPv6 to the Public Subnets, to set the ground work, and go from there.

Enabling IPv6 on your VPC

Regardless of what you need IPv6 for, you need to enable this on your VPC. Just like your IPv4 CIDR that you assign to the VPC, you will need to assign a IPv6 CIDR to your VPC.

⚠️ Difference: Private IPv6 addresses do exist, but you can’t assign them to AWS VPCs. You must use public CIDR ranges ⚠️

In a way, the above does make sense - every device is globally unique, so why would you need to make a private address. Personally, I use internal private addresses to make it easier to remember when connecting to servers, but I am very much an old-school person here, and you should be using name resolution to connect to instances!

Therefore, when you go to assign an IPv6 CIDR range to your VPC, you have one of three options:

For simplicity, I will be using the Amazon-provided CIDR ranges. In the future, I will go over the IPAM-allocated and BYOIP options, but for now these are just additional ways to get an IPv6 CIDR on your VPC.

AWS have access to a fairly large range of IPv6 addresses, this means they can allocate you a unique set of addresses just for you.

⚠️ Terminology: A Network Boarder Group is the name (chosen by AWS) that defines a unique set of Availability Zones or Local Zones where AWS advertises IP addresses ⚠️

When requesting an IPv6 CIDR from AWS, they will need you to select a Network Border Group. This is to ensure that the IPv6 addresses you are receiving can be routed successfully to your VPC. Back to the IPv6 description above, to make routing simpler in the IPv6 world, AWS will route specific ranges to specific groups, and therefore you have to select the right group. Thankfully, as the groups are quite large, for the majority of regions - there is only a single Network Border Group, and Terraform will select this automatically!

Lets start with the vpv.tf that exists in the sample code (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc.tf).

# Creation of the sample VPC for the region
#tfsec:ignore:aws-ec2-require-vpc-flow-logs-for-all-vpcs
resource "aws_vpc" "sample_vpc" {
  cidr_block = var.vpc_cidr

  tags = {
    "Name" = "Sample-VPC"
  }
}

Pretty simple, probably a little too simple! But keep in mind that this is just a sample for now!

Terraform has two resource parameters that will be used to set and assign the IPv6 CIDR to the VPC. assign_generated_ipv6_cidr_block and ipv6_cidr_block_network_border_group. The border group is used a lot more with Local Zones, so we don’t need to worry about this at the moment, but do keep this in mind for more complex deployments.

Just setting the assign_generated_ipv6_cidr_block to true is enough for AWS to assign the VPC a CIDR range. Terraform documentation.

Your file should now look like this:

# Creation of the sample VPC for the region
#tfsec:ignore:aws-ec2-require-vpc-flow-logs-for-all-vpcs
resource "aws_vpc" "sample_vpc" {
  cidr_block                       = var.vpc_cidr
  assign_generated_ipv6_cidr_block = true

  tags = {
    "Name" = "Sample-VPC"
  }
}

With this, your terraform plan should now look like this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_vpc.sample_vpc will be updated in-place
  ~ resource "aws_vpc" "sample_vpc" {
      ~ assign_generated_ipv6_cidr_block     = false -> true
        id                                   = "vpc-0123456789abcdef0"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
        tags                                 = {
            "Name" = "Sample-VPC"
        }
        # (16 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

As you can see with the plan, this will grab some additional details to add to the resource object as attributes to reference later on. This will be key to make your terraform portable, and not hard coded!

Once applied, this will then allocate the IPv6 CIDR to the VPC, and you should then see the following!

Console view of the CIDR allocations on a VPC showing the IPv6 allocation

There we go, we have hit the first step! IPv6 is now enabled on the VPC! As you can see, we have received a /56 block of IP’s. That gives you the room to have a total of 4,722,366,482,869,645,500,000 hosts in your network. I don’t think we will be running out any time soon!

Next step, lets assign to each of the subnets their own CIDR, so that resources in the subnets can assign their own IPv6 address.

Adding IPv6 CIDRs to the public subnets

Just like IPv4 CIDR’s, you can break down the IPv6 range you have into smaller ranges that are all routed internally using the AWS’s VPC networking. With a manual set up you would normally do something like this:

IPv4 VPC CIDR: 192.168.0.0/20
Public Subnet in AZ1: 192.168.10.0/24
Public Subnet in AZ2: 192.168.11.0/24

This is shown in the sample VPC we are using in this post. Simple enough right? Breaking down the CIDR into smaller subnets, and then assigning them to the correct location. It is very much the same with IPv6, but the ranges are just that much larger, that sometimes its best to use an automated method for doing this. What everyone should be doing, is the automatic generation of the ranges for IPv4 as well, which is available in the example!

To do this, we can use a terraform function called cidrsubnet (https://developer.hashicorp.com/terraform/language/functions/cidrsubnet). This function can calculate the subnet addresses within a given IP network block or CIDR, and then be used as a value for a variable in the aws_subnet resource block.

This function takes a bit of getting used to, but here is my trick for understanding how it works!

The function looks like this:

`1`	`cidrsubnet(prefix, newbits, netnum)`

For further details, feel free to read the documentation above, but for our sample we will use the following:

prefix the CIDR range. Available as an attribute as this is generated by AWS. aws_vpc.sample_vpc.ipv6_cidr_block
newbits this is the number of additional bits you need to add to the CIDR prefix, to break the network down. In our case we will use 8 as it is a round number, but you will need to size this to your specifications.
netnum this is the network number assigned to the broken down blocks that you will select for this subnet. It can’t be more than the newbits and you can’t use the same netnum on multiple subnets.

The trick I have used is as follows. newbits is calculated as the difference between the CIDR’s prefix /56 and the network size you need. So in our case I would like to make each subnet a /64 in size. The difference between the two (64 - 56 = 8) means the bit difference is 8. For the moment, I will leave this here, but I will create an article in the future that explains how this works, and why its the number of bits!

For the netnum I tend to visualise it in the diagram - I wanted 4 subnets in my sample, (2 public, 2 private), so I am able to reference the networks created above by their counter, with the first network being 0, with the last network being 3 (as your counter started at 0).

So for our networks, we can enter the values and get the following:

cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 0) # Subnet 1 - x:x::0:0/64
cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 1) # Subnet 2 - x:x::1:0/64
cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 2) # Subnet 3 - x:x::2:0/64
cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 3) # Subnet 4 - x:x::3:0/64

The next step is the parameter for the aws_subnet resource block, which is ipv6_cidr_block. Essentially the same as the cidr_block parameter, but for IPv6!

So for each of our public subnets, lets add this in. In our sample file vpc_subnets.tf (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_subnets.tf), we have two public subnets, a and b. So lets make the change. Below is the example with the new parameters added:

# Public Subnet A
resource "aws_subnet" "public_a" {
  vpc_id               = aws_vpc.sample_vpc.id
  ipv6_cidr_block      = cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 0)
  cidr_block           = cidrsubnet(aws_vpc.sample_vpc.cidr_block, 4, 10)
  availability_zone_id = data.aws_availability_zones.available.zone_ids[0]
  tags = {
    "Name" = "Public-Subnet-A"
  }
}

# Public Subnet B
resource "aws_subnet" "public_b" {
  vpc_id               = aws_vpc.sample_vpc.id
  ipv6_cidr_block      = cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 1)
  cidr_block           = cidrsubnet(aws_vpc.sample_vpc.cidr_block, 4, 11)
  availability_zone_id = data.aws_availability_zones.available.zone_ids[1]
  tags = {
    "Name" = "Public-Subnet-B"
  }
}

Once again, we run our terraform plan and we should get an output similar to this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_subnet.public_a will be updated in-place
  ~ resource "aws_subnet" "public_a" {
        id                                             = "subnet-0123456789abcdefg"
      + ipv6_cidr_block                                = "xxxx:yyyy:zzzz:nnn1::/64"
        tags                                           = {
            "Name" = "Public-Subnet-A"
        }
        # (15 unchanged attributes hidden)
    }

  # aws_subnet.public_b will be updated in-place
  ~ resource "aws_subnet" "public_b" {
        id                                             = "subnet-gfedcba9876543210"
      + ipv6_cidr_block                                = "xxxx:yyyy:zzzz:nnn2::/64"
        tags                                           = {
            "Name" = "Public-Subnet-B"
        }
        # (15 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

The plan shows the addition of the new CIDR block to each subnet, noting the network number changing slightly to accommodate the size we requested. Once applied, we should see these details in the console:

IPv6 CIDR on one of the two sample subnets

Perfect, now let’s launch an EC2 instance and then try and connect to something using the IPv6 address!

The EC2 instance can't connect to an IPv6 address

Well, almost there - we re missing the routing.

Adding IPv6 Routing to the public subnets

Having a look at our existing route tables, we can see that AWS added in the route for the VPC IPv6 CIDR to target the local VPC in the route table, which means it can find resources in the VPC that have the IPv6 address that is within the CIDR. Great for local traffic, but we need to be able to access the outside world using IPv6!

⚠️ Note: While traffic can still route with IPv4, we need to enable routing for IPv6, otherwise any traffic inbound to the server won’t be able to send traffic back. ⚠️

The sample Public-Route-Table doesn't have any IPv6 route to the internet

We can also see this in the sample code too, in the vpc_public_routing.tf file (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_public_routing.tf)

# Primary Sample Route Table (Public)
resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.sample_vpc.id

  tags = {
    "Name" = "Public-Route-Table"
  }
}

# Route entry specifically to allow access to the Internet Gateway
resource "aws_route" "public2igw" {
  route_table_id         = aws_route_table.public_rt.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.sample_igw.id
}

Here we can see the Public-Route-Table resources, but it only shows traffic to the Internet Gateway (IGW) for IPv4 traffic. We will need to add a route to allow IPv6 traffic to hit the IGW.

We can do this by creating a new aws_route resource, but we need to use the destination_ipv6_cidr_block parameter instead.

For the destination though, with IPv4 you could use the block 0.0.0.0/0 to mean “all traffic”. If we were to type the IPv6 version out in full, it would look like 0000:0000:0000:0000:0000:0000:0000:0000/0. Bit of a pain! Thankfully we can apply the short hand rule mentioned at the start, remove all the 0’s, shrink down, and you get quite simply ::/0, which suddenly seems much shorter than the IPv4 version! With this we can add this as the destination IPv6 block.

resource "aws_route" "public2igw_ipv6" {
  route_table_id              = aws_route_table.public_rt.id
  destination_ipv6_cidr_block = "::/0"
  gateway_id                  = aws_internet_gateway.sample_igw.id
}

Once again, lets run the terraform plan and we should get something like this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_route.public2igw_ipv6 will be created
  + resource "aws_route" "public2igw_ipv6" {
      + destination_ipv6_cidr_block = "::/0"
      + gateway_id                  = "igw-0123456789abcdefg"
      + id                          = (known after apply)
      + instance_id                 = (known after apply)
      + instance_owner_id           = (known after apply)
      + network_interface_id        = (known after apply)
      + origin                      = (known after apply)
      + route_table_id              = "rtb-0123456789abcdefg"
      + state                       = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

And once applied, we should be able to see the new route in the routing table!

::/0 has been added to the route table, pointing to the Internet Gateway (IGW)

Jumping back on our EC2 instance, and we get a connection!

We are now able to connect using IPv6 to the outside world!

⚠️ Attention: Inbound traffic to an EC2 instance, for example, will still be protected by its Security Group. The rules in a security group are specific to the IP protocol, so an allow for an IPv4 inbound rule, will only allow that. Make sure that you add any additional IPv6 rules to the security group to permit inbound access to resources in the Public Subnet ⚠️

Adding IPv6 outbound routing to the private subnets

We are getting to the last part of this post about enabling IPv6 on AWS, and we do need to cover the private subnets to complete the sample network configuration. This part will come with a few warnings, but if you are keeping in line with the AWS Well-Architected Framework, this will not be an issue at all!

Thinking back to what we mentioned before, AWS will use global unicast addresses for resources and services in AWS. Meaning, that you do not have a “private” CIDR for your private subnets. As you know, in IPv4 there is the RFP1918 that defines a number of IP blocks, specifically for “internal connectivity”. These IP ranges are not routable on the public internet. This allowed the expansion of devices within a private network, without using up the public internet space. As IPv6 can assign every device on the planet, it makes it easier for us to ensure that the address assigned to a node is unique.

Next we have to look at the definition of what AWS considers a “public” subnet and a “private” subnet. A “public” subnet, is considered such, whereby the route table for the subnet points its outbound internet traffic directly to an Internet Gateway (IGW), and the IGW allows external traffic to route to the subnet. For a “private” subnet, there is no IGW for it to connect to, and you would use a service such as a NAT Gateway (NGW) to connect through to the internet, and as the NAT Gateway doesn’t allow traffic inbound to the network, and the IP address of the node will be a non-internet rotatable IP address, traffic can’t reach the service in the VPC.

This definition sill applies to Private IPv6 subnets, and this is why AWS created the IPv6 Egress-Only Internet Gateway. Much like the IPv4 Internet Gateway counterpart, the IPv6 Egress-Only Internet Gateway is a horizontally scaled, redundant, and highly available VPC component that allows outbound communication for IPv6 traffic to the internet. As this new IPv6 egress-only internet gateway only permits traffic outbound, and not inbound, this will make a subnet private, as traffic can not reach it.

As this is a VPC component, and not a service like the NAT Gateway, you technically only need to deploy one, like the Internet Gateway, and point outbound traffic to the egress-only internet gateway, and it will scale as needed.

With this in mind, lets start by adding in the Egress-only Internet Gateway. This is created by the terraform resource aws_egress_only_internet_gateway (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/egress_only_internet_gateway). This is very similar to the normal Internet Gateway, and even the set up is the same.

Within the vpc_private_routing.tf file (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_private_routing.tf) you will need to add the following resource:

# IPv6 Egress-only Internet Gateway
resource "aws_egress_only_internet_gateway" "sample_ipv6_egress_igw" {
  vpc_id = aws_vpc.sample_vpc.id

  tags = {
    "Name" = "Sample-VPC-IPv6-Egress-Only-IGW"
  }
}

Once again, running terraform plan will output something similar to this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_egress_only_internet_gateway.sample_ipv6_egress_igw will be created
  + resource "aws_egress_only_internet_gateway" "sample_ipv6_egress_igw" {
      + id       = (known after apply)
      + tags     = {
          + "Name" = "Sample-VPC-IPv6-Egress-Only-IGW"
        }
      + tags_all = {
          + "Environment" = "Sandbox"
          + "Name"        = "Sample-VPC-IPv6-Egress-Only-IGW"
          + "Source"      = "Terrform"
        }
      + vpc_id   = "vpc-0123456789abcdefg"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

At this point, I would like to point out - that all the changes we have made so far, have not increased the cost of running! Much like the Internet Gateway, the only cost you have is the outbound traffic. As the Egress-Only Internet Gateway is the same, there is no cost for running this in a VPC other than the outbound traffic you intend to push through it. Unlike the NAT Gateways, which will cost you about $36.50/month, and then for best practice, you would need one in each availability zone, which then adds up. It does make IPv6 only networking in AWS far cheaper than IPv4!

Apply the changes, and the new Egress-only Internet Gateway will be created.

Adding IPv6 CIDRs to the private subnets

The next step, is that we need to add the CIDR blocks to the private subnets. This is pretty much identical to the public subnet addition, in fact, it is identical. This is due to there being no difference between public blocks and private blocks within the VPC IPv6 range. So for speed, we just repeat the same.

Head back to our sample file vpc_subnets.tf (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_subnets.tf), and look for the two “private” subnets. Add in the ipv6_cidr_block for each of them, based off the next two blocks calculated earlier:

# Private Subnet A
resource "aws_subnet" "private_a" {
  vpc_id               = aws_vpc.sample_vpc.id
  ipv6_cidr_block      = cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 2)
  cidr_block           = cidrsubnet(aws_vpc.sample_vpc.cidr_block, 4, 12)
  availability_zone_id = data.aws_availability_zones.available.zone_ids[0]
  tags = {
    "Name" = "Private-Subnet-A"
  }
}

# Private Subnet B
resource "aws_subnet" "private_b" {
  vpc_id               = aws_vpc.sample_vpc.id
  ipv6_cidr_block      = cidrsubnet(aws_vpc.sample_vpc.ipv6_cidr_block, 8, 3)
  cidr_block           = cidrsubnet(aws_vpc.sample_vpc.cidr_block, 4, 13)
  availability_zone_id = data.aws_availability_zones.available.zone_ids[1]
  tags = {
    "Name" = "Private-Subnet-B"
  }
}

Run the terraform plan and apply the changes to assign the blocks to the subnets.

Adding IPv6 Routing to the private subnets

This is where the limitations with the IPv4 NAT Gateway makes the changes for a private subnet add additional operational overhead to the work needing to be completed. In our sample code, we created two route tables for the private subnets, one for each availability zone, so that traffic within the subnet routes through the NAT Gateway for that availability zone.

This means, rather than like the public subnet, where we need to add just a single route in terraform. We have to add two and attach them to both route tables.

If you were creating an IPv6 only VPC, then you could reduce the work and have a single route table that works for both availability zones! However, we will look at this at a later date.

This time, we need to head to the vpc_private_routing.tf file (https://github.com/mystcb/ipv6-on-aws/blob/main/01-sample-vpc/vpc_private_routing.tf) again, and we need to add in the two new routes.

If you look at the file, you will see that for the IPv4 NAT Gateway, there is a specific parameter that you use to tell terraform to issue the right API command to AWS to add the route, which you can see here:

# Route for Subnet A to access the NAT Gateway
resource "aws_route" "private2natgwa" {
  route_table_id         = aws_route_table.private_rt_a.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.sample_natgw_subnet_a.id
}

The nat_gateway_id will be specifically used to pointing to the ID of the NAT Gateway within the availability zone. Much like in the public subnet you would use gateway_id for the Internet Gateway, you can use the egress_only_gateway_id for the IPv6 traffic.

Therefore, we will need to add 2 new blocks to the terraform file, to add in the route ::/0 to point to the Egress-only Internet Gateway.

⚠️ Attention: Remember, for IPv6 routes, you will need to use the destination_ipv6_cidr_block as part of the route table resource ⚠️

# Route for Subnet A to access the Egress-Only Internet Gateway
resource "aws_route" "private2ipv6egressigwa" {
  route_table_id              = aws_route_table.private_rt_a.id
  destination_ipv6_cidr_block = "::/0"
  egress_only_gateway_id      = aws_egress_only_internet_gateway.sample_ipv6_egress_igw.id
}

# Route for Subnet B to access the Egress-Only Internet Gateway
resource "aws_route" "private2ipv6egressigwb" {
  route_table_id              = aws_route_table.private_rt_b.id
  destination_ipv6_cidr_block = "::/0"
  egress_only_gateway_id      = aws_egress_only_internet_gateway.sample_ipv6_egress_igw.id
}

For one final time, run the terraform plan and you should see an output similar to this:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_route.private2ipv6egressigwa will be created
  + resource "aws_route" "private2ipv6egressigwa" {
      + destination_ipv6_cidr_block = "::/0"
      + egress_only_gateway_id      = "eigw-0123456789abcdefg"
      + id                          = (known after apply)
      + instance_id                 = (known after apply)
      + instance_owner_id           = (known after apply)
      + network_interface_id        = (known after apply)
      + origin                      = (known after apply)
      + route_table_id              = "rtb-0123456789abcdef1"
      + state                       = (known after apply)
    }

  # aws_route.private2ipv6egressigwb will be created
  + resource "aws_route" "private2ipv6egressigwb" {
      + destination_ipv6_cidr_block = "::/0"
      + egress_only_gateway_id      = "eigw-0123456789abcdefg"
      + id                          = (known after apply)
      + instance_id                 = (known after apply)
      + instance_owner_id           = (known after apply)
      + network_interface_id        = (known after apply)
      + origin                      = (known after apply)
      + route_table_id              = "rtb-0123456789abcdef2"
      + state                       = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Remember, that we are adding two routes, as we have two route tables to make the change to. Apply the changes, and you should see the new route in the route tables:

New route pointing to the new Egress-Only Internet Gateway VPC resource

So, jumping on an EC2 instance in the private subnet (you can see on the command line that the instance is in the 192.168.12.0/24 subnet), you can see we have complete outbound access on IPv6!

Private Subnet EC2 instance with connectivity through the Egress-Only Internet Gateway

The final outcome

So you finally did it, you have enabled IPv6 on your VPC. If all went to plan, you should have something that looks like this:

Sample VPC with IPv6 enabled

To make things a little more simpler, you can also check out the 02-vpc-with-ipv6 folder in the sample code (https://github.com/mystcb/ipv6-on-aws/tree/main/02-vpc-with-ipv6), which will produce the same output as the diagram, and if you followed the changes!

Summary and round up

Thank you for getting this far! There is a lot here, but hopefully I have been able to show you how to add IPv6 to an AWS VPC using Terraform. However, this is only the beginning. Throughout my time working with IPv6, there will always be different issues to trip you up, and there are far more features than just a simple VPC!

In a future post I hope to show you the following:

Adding IPv6 to an existing EC2 instance inside a VPC without rebuilding it. A little harder than I expected with Terraform, but I did find a way around it!
IPv6 only VPCs! Much cheaper to run that a normal IPv4 VPC, but at what “cost”, adoption of IPv6 is still quite low, but there are a number of design patterns that mean an IPv6 VPC might work better for some situations.
Going into IPv6 in more detail. This is only the very basic information on IPv6, and a lot of what I have mentioned, does come with caveats! I will hope to explain these in more details later on
Bits? Why does one number mean something else, and why on earth do we count in bits? - Hope you have learnt binary for this one!

Thank you again, and feel free to send me any questions, or help me with any corrections to this post as well! Hopefully, I will see you in a future post!

P.S. The final cost of me running this for the creation of the post, was mainly the NAT Gateways! Everything else was free! It only cost me $1.50 total! Always make sure you run terraform destroy at the end!

P.P.S. Happy Birthday to me!