Routing on Colin Barker

AWS Direct Connect Allowed Prefix lists - My "gotchas" with it!

Mon, 21 Oct 2024 14:52:00 +0000

Header photo by Jan Huber on Unsplash

What is AWS Direct Connect?

In this world of cloud technologies, and the idea that the cloud can solve all your problems, there is always a need to have some level of connection from an on-premise, or data centre location into AWS. Typically, the use of a VPN can be enough for most organisations, and it is a lot cheaper than AWS Direct Connect however, there will always be a wider security policy, or additional protections you need on your network. This is where AWS Direct Connect comes in.

Unlike a VPN connection, which creates a private, encrypted tunnel between different networks and AWS, AWS Direct Connect can be seen in the most simplest terms, as plugging a network cable from a switch in your racks into a switch inside AWS that is connected directly into an AWS network. A layer 2 “wired” connection that you can push traffic you need through it. The connection never transits through the public internet, and is completely private (to a degree). It can reduce the number of hops over different internet routers to get to your AWS network, reducing the latency and improving the available bandwidth to and from your AWS resources, and depending on the size of the pipe you request from AWS, it could be faster than what is currently capable over a standard Site to Site VPN connection.

It uses a networking standard called 802.1Q VLAN tagging to be able to segment the traffic, by tagging traffic that transverses the network, switches can ensure that only the right ports see the right traffic. This is a very similar concept to the VLANs that you may be familiar with from your home network, same technology, just in a different context.

This post isn’t meant to be a complete re-hash of the documentation! If you would like to know more, then feel free to look at the AWS Direct Connect documentation. Instead, I will go through some of the gotchas that I have encountered with AWS Direct Connect while building a network for a customer.

Allowed Prefixes for AWS Direct Connect gateways

There is a whole page on the AWS documentation called Allowed prefixes interactions specifically for this, so I will talk about the specific gotcha that I encountered. The allowed prefixes act differently depending on the type of association you have linked your AWS network up to Direct Connect with. Virtual Private Gateway (VPG) or Transit Gateway (TGW), the list changes from a filter to a whitelist of what can be advertised over Direct Connect, and in the case I worked with, in both directions.

Example of an AWS Direct Connect connection between a Transit Gateway and a Virtual Private Gateway

Firstly, this is just an example of a wider customer example! It wasn’t quite set up this way, but this is summarised to show the potential of two different ways to connect your network to AWS. In reality, the customer did have something similar setup, but connected to two different types of setup, that we came in to resolve for them. For this we shall look at the Allowed Prefixes list across both association types.

Where to find the Allowed Prefixes list

This one caught me out when looking around the interface, if you have never had to use AWS Direct Connect before, while you know it might exist from training, locating the list took a few minutes to find! On many occasions during customer calls I got myself lost looking for this, mainly because - it’s available in two locations!

Location of the Allowed Prefixes from the Direct Connection Gateways UI

Location of the Allowed Prefixes from the Transit Gateways UI

Both locations will send you to the same place, so don’t worry that you are editing one and need to amend the other.

NOTE: Just like with any other BGP connection, making a change to this list will reset the BGP connection and re-advertise the prefixes in the list. This normally isn’t an issue, but sometimes this can cause a connectivity break if there is an issue somewhere else in the BGP sessions that exist. Just be aware of this if you need to make a change. Include it in any change request, or process you have to amend the list.

An example setup

Simple example of two different groups of CIDRs, can we work out what will be advertised on site?

In this example, we have listed essentially what the VPC’s are advertising into the different associations. This hasn’t covered the lower level examples of how this connects in, we shall assume that prorogation has been enabled properly on each, and both the Transit Gateway and the Virtual Private Gateway are receiving the correct prefixes.

Allowed Prefixes list with a Transit Gateway Association

Starting with an AWS Transit Gateway association, we had traffic being advertised over BGP to AWS, and the VPC’s being pushed back through to on-premise. We would need to be able to advertise network from the VPC connections, and On-Premise networks into the Transit Gateway. Remember, that essentially Direct Connect is one “router” and the Transit Gateway is another, linked with a “virtual cable”. The allowed prefix list in this case acts as both a filter and an announcer right in the middle of the two, but controlled from the Direct Connect side. This list acts as the CIDR’s that get advertised into the Transit Gateway. The documentation calls it out on this specific section, but it caught me out with my customer!

For this example, we are going to use an allowed prefix list that looks like this:

10.0.0.0/16
172.26.0.0/20
192.168.0.0/20
100.70.0.0/24

With this prefix list and the above listed AWS advertised prefixes, this will show you what gets advertised on to the on-premise network. This works the opposite way around as well, for on-premise networks advertised into AWS however, for this example we will just go one way.

AWS Advertised Prefix	Allowed Prefix Entry	On-Premise Received Prefix	Notes
10.0.0.0/16	10.0.0.0/16	10.0.0.0/16	Simple example, what was advertised is what was received
172.26.0.0/16	172.26.0.0/20	172.26.0.0/20	Here we have a larger CIDR in AWS, but the prefix is smaller in the allowed prefix list, so the smaller prefix is what is received
192.168.0.0/24	192.168.0.0/20	192.168.0.0/20	Here we have a smaller CIDR in AWS, but the prefix is larger in the allowed prefix list, so the larger prefix is what is received
100.64.0.0/24	None	None	An example of the filter element working, while we have configured the AWS side to advertise the prefix, it is not allowed to be advertised over Direct Connect to on-premise
None	100.70.0.0/24	100.70.0.0/24	This is where it can get a little complex, we have added the allowed prefix, but it isn’t being advertised on AWS -or- on premise. In this instance both sides of the association will receive the prefix, even though there is no network

GOTCHA: Be careful when using the allowed prefix list with a Transit Gateway association, not to try and open up wider CIDR ranges than you need to, as this can have an unindented effect on the traffic that is advertised into the Transit Gateway and on premise.

Allowed Prefixes list with a Virtual Private Gateway Association

Moving onto the Virtual Private Gateway association, this connection uses pure filtering. If you are on the filter list, then you will be allowed to advertise, if you are not, then you can’t. The CIDRs advertised must be exact otherwise the filter will block it. This list will not actively announce any CIDR in the list, so you can’t use it to advertise non-existent or wider ranges to make “administration” easier later on! The AWS Documentation also calls this out but for me, this is where I was also caught out.

For this example, we are going to use an allowed prefix list that looks like this:

10.100.0.0/24
172.16.0.0/20
192.168.0.0/20

With this prefix list and the advertised routes that exist in the AWS side, we will show you what gets advertised on the on-premise network. Just like before, this works the opposite way around as well, for on-premise networks advertised into AWS however, for this example we will just go one way.

AWS Advertised Prefix	Allowed Prefix Entry	On-Premise Received Prefix	Notes
10.100.0.0/24	10.100.0.0/24	10.100.0.0/24	Simple example, what was advertised is what was received, filter is allowed
172.16.0.0/16	172.16.0.0/20	None	We h ave a wider CIDR in AWS, but the filter is of a smaller range, therefore this does not get advertised into the on-premise network
192.168.20.0/24	192.168.16.0/20	192.168.20.0/24	Here we have a smaller CIDR in AWS, and the filter is for a wider range, as the smaller prefix is within the larger prefix on the filter list, it will allow it through to the on-premise network
100.74.0.0/24	None	None	Simple example, this CIDR is not in th prefix list, so it is not allowed to be advertised

GOTCHA: Here we can see that the filter list is not the same as before, different ranges get advertised in different ways. While this is talked a lot in the AWS documentation, getting the experience of using this with AWS Direct Connect is a little harder due to it’s specific usage and adoption within different customers.

Summary

The final outcome of what would be received by the Customer Gateway

Sometimes it can feel counter intuitive on how the two types of allowed prefix lists work, but they are important in knowing the best way to configure a large organisations connection into AWS. Being able to see this in person is very hard to see due to the adoption of AWS Direct Connect, so keep this in mind the next time you run into an odd routing problem with the use of AWS Direct Connect.

Zero Trust Network Router on AWS

Fri, 12 Apr 2024 18:41:00 +0000

Header photo by Sander Weeteling on Unsplash

What is a “Zero Trust Network”

To put plainly, this is a network that is created that by default has Zero Trust within it, it is based off the idea of a Zero Trust security model which is a specific type of implementation used across different networks.

The main concept behind the zero trust security model is “never trust, always verify”, which means that users and devices should not be trusted by default
― Zero trust security model, Wikipedia

So, how does this apply to networking? Well the main concept here is, that you create a network that by default disallows traffic that isn’t known, verified, or wanted. This can be expanded, to then have all your devices connected into a single network, that by default, only allows traffic to pass between each node if you accept it. This is a concept that is often referred to as a “Zero Trust Network”.

Example of a simple network with switches and WiFi connection point

Starting with a basic scenario, above we have a simple office, it has a Data Room with a load of servers, with a couple of Virtual Machines, some Head office computers, a physical server, and a mobile phone connected to a WiFi network. Using this as a base, everything had to be in the same place, but it was all on the same network - devices would talk to each other through the switch, and traffic didn’t need to jump over anything other than the switch (excluding the phone!).

As the business expanded, it migrated the on-premise data room to a data centre, the Head Office was moved to a new location, and people had the option of working from home. Additionally, the CTO has asked, “We need to move to AWS”. Networking before this would have been quite complex to set up.

This is where a Zero Trust Network comes into play.

Depiction of a Zero Trust Network solution, with AWS hooked in

Let’s take a look at the above example. Even though the Head Office move was completed, the data centre had been set up, and there were workloads in AWS, we can see here - the network is still directly connected. With this scenario, each segment is part of the wider network, rather than using a VPN connection to route traffic between them, you can see that it is possible to just talk from one side to another, with the only hop being the Zero Trust network in the middle. Think of the provider as essentially a switch!

In this post, I will concentrate primarily on the Zero Trust Appliance in AWS, and how we can connect to a Zero Trust Network.

Introduction to Tailscale

Tailscale is one of the many different Software Defined, Zero Trust networking solutions that exist today. Many different providers have different ways of implementing their solutions, but they all are based on the same simple premise, “never trust, always verify”. Tailscale has multiple methods for adding devices to the network, by default, the Access Control Lists (ACLs) will not permit traffic between different devices unless explicitly states in the configuration of the ACL.

For anyone looking at behind the scenes of the technology used at Tailscale, I would recommend “How Tailscale Works” by Avery Pennarun who wrote how the data plan works, and a couple of examples as to why traditional VPNs might cause latency or even general issues in network connectivity.

Our example, Tailscale to connect an Office to AWS

Simply put, Tailscale will act as our “switch” and set up the point-to-point network between an EC2 instance, and an office where they might have several desktops.

For this post today, I will concentrate specifically on the Appliance that will sit in AWS, and how we would configure this for a customer.

Building a Zero Trust Network Router on AWS

For this, we will be using a specific set of tooling:

Terraform v1.7.5 - An infrastructure as code tool
AWS Provider - The translator between Terraform and the AWS API
Tailscale Provider - The translator between Terraform and the Tailscale API

Each “device” that is connected to Tailscale will need to be authenticated and approved before the device will be given access to the network. Even if there is an ACL in place that permits the access, the approval must occur. This can cause an issue when working with Infrastructure as Code (IaC), as the process would need to be automated. This can be overcome by generating a Tailnet Auth Key that is used specifically for the launching of the instance, and more specifically allowing your device to be added with “pre-approval”. For this, we will create a “tailnet key” resource.

# Create an authorization token for the Tailscale router to add itself to the Tailnet
resource "tailscale_tailnet_key" "tailnet_key" {
  preauthorized = true    # Set to true to allow the pre-approval of the device
  expiry        = 7776000 # Time in seconds
  description   = "Tailnet key for the server"
}

Simply put, this generates the key as a resource, and then Terraform knows what the key is, and any other properties that will have been shown by the API call to create the key.

With this key, the next step would be to install the Tailscale client onto the EC2 instance and make it into a router for any services within AWS. This would need to be done in a few stages when working with IaC, so we should start right at the beginning.

Tailscale offers a very comprehensive guide on how to install the tailscale client on a vast number of devices, including Linux, macOS, Windows, iOS, Android, and even down to Amazon Fire Devices, and Chromebooks too! What we are doing is creating a Subnet Router.

The Subnet Router is slightly different on a Zero Trust Network like this, while it is usually recommended to install the client on every single server, client, and virtual machine in the organisation, sometimes - it’s not needed. Even more so for organisations that use the Cloud, and have a LOT of ephemeral devices, and where the Software Defined Network of the Cloud Provider already has the security in place that keeps your network Well-Architected.

Within Terraform, there is a function called templatefile that can be used to generate strings or blocks of text that can use variables that are generated from within Terraform and then used within the string or block of text. Here, we are using the output of the tailscale_tailnet_key resource above, and pushing the generated key value into the user_data for an AWS EC2 Instance, to run a script on the first run. Below is the template used to install Tailscale.

#!/bin/bash

## Set the hostname of the server
hostnamectl hostname ${hostname}

## Ensure that the server is up to date with all the current packges
DEBIAN_FRONTEND=noninteractive sudo apt update -y
DEBIAN_FRONTEND=noninteractive sudo apt upgrade -y

## Enable IP Forwarding on the router to ensure that packets will flow as required
echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

## Install Tailscale from the source
curl -fsSL https://tailscale.com/install.sh | sh

## Start Tailscale with the authorisation key to add it to the network
sudo tailscale up --authkey=${ts_authkey} --advertise-routes=${local_cidrs} --accept-routes

There is quite a bit happening in the script, it can be summarised as follows:

Set the hostname for the instance - While this is an EC2 instance, and usually in the cloud you would probably use more ephemeral devices, a Subnet Router will need to act more like a “pet” so that Tailscale sees this as an appliance object. Setting the hostname will mean that it is recognisable as to which host this is within your Tailscale network
Update the instance using apt - Pretty simple, make sure the instance is running the latest updates and patches before continuing!
Set IP Forwarding on the device - This is a key step, especially so in Linux, as this is to enable the EC2 instance to take traffic that it receives and forward it on. This specific setting tells the networking device at a kernel level to route traffic through it, as by default this is switched off.
Install Tailscale - The primary install of Tailscale, taken from the latest version on the Tailscale site.
Startup tailscale WITH the keys - Here we finally get to see where our tailscale key will be used - using the up function, we can set the authkey generated before, as well as which routes to advertise. We will go into this in a second, but for now, this is where the key will go.

Simple enough script, now we have to move on to creating the EC2 instance that will run

# Use a data object to get the latest version of Ubuntu
data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"]
}

# Build the Tailscale router using Ubuntu 22.04 LTS
resource "aws_instance" "this" {

  # General setup of the instance using the Ubuntu AMI
  ami                     = data.aws_ami.ubuntu.id
  ebs_optimized           = true
  instance_type           = "t4g.small"
  disable_api_termination = true

  # Key for SSH Access (if required)
  key_name = try(var.ssh_key_name, null)

  # Run the user_data for this instance to install Tailscale
  user_data = templatefile("${path.module}/templates/tailscale-install.sh.tpl",
    {
      ts_authkey  = tailscale_tailnet_key.this.key
      hostname    = var.hostname
      local_cidrs = var.local_cidrs
    }
  )

  # Networking Settings
  source_dest_check      = false # Disabled to allow IP forwarding from the network
  private_ip             = var.private_ip
  ipv6_addresses         = var.ipv6_addresses
  subnet_id              = var.subnet_id
  vpc_security_group_ids = [aws_security_group.this.id]

  .... (snipped)

}

A lot happening in this section, but in this example, we are using AWS Graviton (ARM-based) instances, as they are known to be a lot more efficient than other processor types, they can be cheaper, but also currently on AWS’s Free Tier for the time being!

The data object for the aws_ami - This will look for the latest version of the Ubuntu 22.04 image that exists in the Canonical account. Note the arm64 element of the filter string to look for the ARM version of the AMI.
The aws_instance resource to generate the subnet router - The standard for any EC2 instance, the primary resource with all its configuration
The general setup of the EC2 resource - Here we are using the data.aws_ami.ubuntu.id to ensure the right AMI is set, additionally making sure that api_termination has been configured, as we don’t want someone accidentally deleting the instance!
An SSH key - Some people might want to ensure they have an SSH key so they can log into the instance, in several cases this might not be needed.
User data template - This part is where we are taking the template bash file above, and taking the variables it knows about and entering details of what Terraform knows. Here we can see the ts_authkey variable is being set to the previously created tailscale_tailnet_key resource. Additional settings are also entered here.
Network Settings - This one contains one key variable that MUST be set for any EC2-based router that is created. The source_dest_check variable must be set to false. The source/destination checking is on by default, and within the software-defined VPC network on AWS, it will ensure that the traffic that sees, is the traffic for it - when it acts as a router, it will expect to see traffic pass through it from different devices that it needs to forward on. This check is disabled to allow this to happen. Without it, there is no way for traffic destined for another part of the network to flow to the EC2 instance.

Once this resource is launched, then it will appear in your Tailscale network, be approved, and should also be able to route traffic to and from your AWS network.

Some other elements that make up the design for this router, for example, you will need Security Groups set up to ensure traffic is allowed inbound and outbound to the EC2 instance and networks. To make this whole process easier, I created a Terraform Module that does all of this for you!

https://github.com/mystcb/terraform-aws-tailscale-router

In Part 2 of this blog post series, I will go into how to use this module to create a Multi-AZ version of this set up, and also include the changes I will be made to the module to enable Instance Recovery if the Operating System stops responding.

Examples of other Zero Trust Networking Solutions

- CloudFlare Zero Trust - This has more recently had an update that will allow access through one of its WARP clients. This is still in beta so one to keep an eye on. - Enclave - This was my first ever experience with Zero Trust networking, so I can’t not name-drop this one! While personally, I don’t use it anymore, it was here I learnt the basics of the Zero Trust network before moving myself to Tailscale.