Iam on Colin Barker

GitHub Actions and OIDC Update for Terraform and AWS

Sun, 12 Jan 2025 14:32:19 +0000

Header photo by Roman Synkevych 🇺🇦 on Unsplash

NOTE: This blog posts follows on from my original post back in 2023 where I first set this up. You can find the original blog post GitHub Actions using AWS and OpenID by using the link.

Before we begin

Please make sure to read my original blog post, it has been updated with the new steps, but this post goes on to specifically call out the changes that allowed this to happen.

What changed?

A few months beyond my original blog post, GitHub and AWS updated the way that they authenticate between each other, by removing the need to pin the certificate thumbprint as part of the OIDC authentication process. GitHub Blog Post. AWS added GitHub to one if its many root certificate authorities (CAs), meaning GitHub can update their authentication certificate without the need for us to update the thumbprint that existed in the setup step of the OIDC setup. AWS Documentation

For a while the AWS SDK and the Terraform AWS Provider had not been updated, so still required a thumbprint to be added to the OIDC provider to allow Terraform to create the resource. Thankfully, in December, AWS updated their main Go SDK, which allowed the owners of the Terraform AWS Provider to make their change to make the Thumbprints Optional.

NOTE For existing implementations, this might be harder to work in, due to a quirk with the AWS API and the way Terraform works. There is a bug issue open for this. Essentially Terraform will not update the API if there is no default setting, clearing out the Thumbprint list with no default means Terraform will not update the API, so the thumbprints are not removed. As the default behaviour for AWS OIDC configurations that are trusted by AWS means that it ignores the thumbprint list, this should not cause any issues with existing OIDC setups. This is only an issue with Terraform, and not the AWS API.

Creation of the OpenID Connect Provider

Setting up the Identity Provider (IdP) will need to be the updated. The walkthrough documentation, has also been updated:

The provider URL - in the case of GitHub this is https://token.actions.githubusercontent.com
The “Audience” - which scopes what can use this. Confusingly in Terraform this is also known as the client_id_list.
You no longer need to generate the thumbprint

Adding the resource in Terraform

Now that we have the only two bits we need, it is easy enough to amend / change your Terraform code to set up the OpenID Connect Provider in IAM. This code example below has been updated to use the only bits you need, removing the need for the old thumbprint_list.

resource "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"
  client_id_list = [
    "sts.amazonaws.com"
  ]
}

Everything else inside the original blog post still stands.

What about other OpenID Connect Providers?

There are only a limited number of providers that AWS have on their root certification CA approved list. Therefore you will need to read the documentation on each provider to find out. If you are still using another OIDC provider with AWS, and need to use the thumbprint, then the usual steps still apply. Below is an updated version of the steps needed for a 3rd party OIDC provider.

You will need to ensure you have the following

The provider URL - this is normally supplied by your third party, and should be a public address
The “Audience” - which scopes what can use this. Once again in Terraform this is also known as the client_id_list.
The Thumbprint of the endpoint - This one is the tricker one, as you will need to generate this yourself.

Generating the thumbprint

For this example, you will need to ensure that you have downloaded the OIDC provider certificates, or have them to hand to generate the thumbprint. Ideally, if you can get the thumbprint from the provider, that would mean you can skip all the steps.

If you know the url that needs to be used, then you can use the following openssl command like before.

`1`	`openssl s_client -servername tokens.endpoint.oidc.provider.com -showcerts -connect tokens.endpoint.oidc.provider.com:443`

Grab the certificate shown in the output, you will see this starting with -----BEGIN CERTIFICATE-----, then place this content into a file. For this demo, I will use openid.crt.
Use the OpenSSL command again to generate the fingerprint from the file created above.

`1`	`openssl x509 -in openid.crt -fingerprint -sha1 -noout`

Which should output the fingerprint. Strip away all the extra parts, and the : between each of the pairs of hexadecimal characters, and you should end up with something like this:

`1`	`6938fd4d98bab03faadb97b34396831e3780aea1`

⚠️ Note: You will need to make the letters lowercase, as Terraform is case sensitive for the variable we need to put this in, but AWS is not case sensitive, so it can sent Terraform into a bit of a loop. ⚠️

Adding the resource in Terraform

With all of this, you can now start to put together the Terraform elements. We can use the Terraform resource for setting up the OpenID Connect Provider in IAM. Below is a code example, using the information gathered from the documentation and the thumbprint generation, and all placed into a single resource object.

resource "aws_iam_openid_connect_provider" "other_oidc_provider" {
  url = "https://tokens.endpoint.oidc.provider.com"
  client_id_list = [
    "sts.amazonaws.com"
  ]
  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1"
  ]
}

Round up

While this is a shorter than normal blog post, this should hopefully show the differences between the two types now. Once we get an update on how the bug regarding existing OIDC setups work, I am sure I will post another update!

Any questions, please let me know!

GitHub Actions using AWS and OpenID

Tue, 28 Feb 2023 16:29:20 +0000

Header photo by Roman Synkevych 🇺🇦 on Unsplash

Edited: 12th Jan 2025 - The AWS SDK and the Terraform AWS Provider was updated to make the thumbprint optional! See my latest blog post here for more information.

Edited: 18th Sept 2024 - As AWS no longer require you to use a thumbprint when setting up the OIDC connection the thumbprint section is no longer needed, however the Terraform Provider still has an open bug for this. Hopefully, once this issue is closed I will remove the section completely!

What is OpenID?

Always a good start, understanding what the key component to this whole post is! As always though, I will reference Wikipedia

OpenID is an open standard and decentralized authentication protocol promoted by the non-profit OpenID Foundation. It allows users to be authenticated by co-operating sites (known as relying parties, or RP) using a third-party identity provider (IDP) service…
― Wikipedia, OpenID

Simple right! Well the technical details of how OpenID works is probably for a much more in-depth specific technical blog for this technology, but the one thing that we need to understand here is that OpenID allows users to be authenticated using 3rd party identify providers. In the case of AWS, their OpenID Connect set up would allow a service in GitHub to authenticate to AWS, and through the IAM system, assume a specific role.

The question is, why go to the effort to set this all up when a simple Access Key/Secret Key combination would work? Well, as you know from AWS Well-Architected best practices, you should always use temporary credentials over static ones. With the assumption of an AWS role, it uses temporary credentials. With OpenID Connect-compatible identity providers, such as GitHub, you would need to set this up using a Web Identity source. With this post, I will show you how I set this up for this blog (and for the Tokonatsu website!)

GitHub Actions using IAM Access Keys

This is where I started, as you can see below GitHub has my secrets for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, statically set quite a while ago!

IAM Access Key and Secret Key statically used

Within my GitHub actions pipeline, I am using the Configure AWS Credentials action from the GitHub Marketplace to configure the secrets for use in the pipeline.

- name: Configure AWS Credentials
  id: aws-credentials-configure
  uses: aws-actions/configure-aws-credentials@v1
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: ${{ secrets.AWS_DEFAULT_REGION }}

If you want to see the whole file, then there is a link here to the build-and-deploy.yaml (prior to the changes) file on GitHub.

As you can see, it is pretty simple, and it worked. The IAM user that owned those keys was happy to sit there and allow the pipeline to access the service. However, that IAM user is still static, and the keys will need to be manually be rotated. From a security stance, anyone that found that key out would be able to do as much as the pipeline could do to my website. As I use aws s3 sync to copy the website up, while also using the -delete parameter, it means that I need delete access for this IAM user! Not really the best!

By using Terraform, I was able to set up all the correct access and switch my pipeline over to using role assumption, thus temporary credentials. GitHub do provide a walkthrough to set up the OpenID Connect, which is what I based this configuration on. Along with the Terraform Documentation hopefully, this will help you with your journey!

Creation of the OpenID Connect Provider

Setting up the Identity Provider (IdP) will need to be the first step. This action will create a description in IAM of the external IdP and establishes the trust between your account and the organisation, in this case GitHub. This step requires just a few options, of which can be harder to get.

Using the walkthrough documentation, we can see that the following is required:

The provider URL - in the case of GitHub this is https://token.actions.githubusercontent.com
The “Audience” - which scopes what can use this. Confusingly in Terrraform this is also known as the client_id_list.
The Thumbprint of the endpoint - This one is the tricker one, as you will need to generate this yourself.

Generating the thumbprint

⚠️ This section is no longer required - AWS and GitHub no longer require the setting of thumbprints between them however, the Terraform Provider for AWS still has an open bug as it still requires a value to enter this in. For the best option, just enter in the value 00000000000000000000000000000000000000000.

This part wasn’t as clear in the GitHub documentation, but I went over to the AWS Documentation which gave me instructions on how to generate the thumbprint. You would need a copy of the openssl CLI o be able to do this, but the quickest way is as follows:

Use the OpenSSL command to check against the provider URL to get the certificate.

`1`	`openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443`

Grab the certificate shown in the output, you will see this starting with -----BEGIN CERTIFICATE-----, then place this content into a file. For this demo, I will use github_openid.crt.
Use the OpenSSL command again to generate the fingerprint from the file created above.

`1`	`openssl x509 -in github_openid.crt -fingerprint -sha1 -noout`

Which should output the fingerprint. Strip away all the extra parts, and the : between each of the pairs of hexadecimal characters, and you should end up with something like this:

`1`	`6938fd4d98bab03faadb97b34396831e3780aea1`

Adding the resource in Terraform

resource "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"
  client_id_list = [
    "sts.amazonaws.com"
  ]
  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1"
  ]
}

This seems simple enough, but this is just authorising GitHub to be a trusted source, and that identities from GitHub can be used to authenticate against AWS. The IAM role set up is where the main bulk of granting access is completed.

Creating the IAM Role

IAM Policy - Bucket Access

Before access can be granted to the GitHub Actions pipeline, we will need to create a policy that defines the access that the role will have. There isn’t much in the way of any difference to this part than creating any other policy however, for this blog and the Tokonatsu website, we need some additional permissions outside of the simple s3:PutObject.

In the example below, as my S3 bucket is also a resource in Terraform, you can see how I have pulled the bucket ARN for the resource from the outputs of the S3 bucket creation. It is normally best practice to reference variables and other outputs rather than using strings.

# tfsec:ignore:aws-iam-no-policy-wildcards
data "aws_iam_policy_document" "website_colins_blog_policy" {
  version = "2012-10-17"

  statement {
    effect = "Allow"
    resources = [
      aws_s3_bucket.website_colins_blog.arn,         # or "unique-bucket-name-for-site"
      "${aws_s3_bucket.website_colins_blog.arn}/*"   # or "unique-bucket-name-for-site/*"
    ]

    actions = [
      "s3:DeleteObject",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:PutObject"
    ]
  }
}

⚠️ Note: As I use tfsec to keep an eye on my Terraform, it attempts to look at the policy and look for anything that might be considered an issue. On the first line you can see the tfsec:ignore:aws-iam-no-policy-wildcards comment, which means tfsec will ignore that rule when it checks my Terraform. As we need to give this role access to do the listed actions on all objects in the bucket, a wildcard is easier. Hence the rule to stop the error from showing up. ⚠️

IAM Policy - Role Assumption

As we will be using rule assumption, there needs to be an additional policy document created that will be added to the Role, that can tell it who can assume the role. You might have seen a similar version when using sts:AssumeRole as an action, for principles that cover other accounts as an example. With the sts:AssumeRoleWithWebIdentity what we are telling AWS tha the role assumption needs to happen if they have a “web identity”, one of the external providers.

The primary element to ensuring the right IdP is used when looking for which identity to grant access too, we need to reference the ARN of the OpenID Connect Provider added earlier. As the Terraform resource we used above was identified with aws_iam_openid_connect_provider.github, we can use one of its attributes to programmatically add the ARN to the principles. We will need to specify the type in Terraform as “Federated” as well.

The last two bits are condition blocks and these are unique to the GitHub OpenID Connect set up. There is a little more detail on the GitHub Actions: OpenID Connect in AWS page. Using this page, I am adding two “StringLike” tests to the role, that looks for two specific variables:

token.actions.githubusercontent.com:sub - Which is used to specify which repo’s GitHub actions are allowed access.
token.actions.githubusercontent.com:aud - That ensures that AWS’s STS service is the one that is requesting the identity type, and no others.

The token.actions.githubusercontent.com:sub in my actual repo it says that any repo in my personal space mystcb called blog using any branch, can access. While this is very open, my personal blog only really can be edited by myself, and I would want the role to also activate on any test branches as well. This is shown on my personal Terraform as repo:mystcb/blog:*. The example I have below is more specific, and only allows access if the GitHub action is working from the main branch. What I wanted to show here is a secure version, but also showing how you can add wildcards to the value to cover more branches if need be.

data "aws_iam_policy_document" "website_colins_blog_role_assumption" {

  version = "2012-10-17"

  statement {
    effect = "Allow"

    actions = ["sts:AssumeRoleWithWebIdentity"]

    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:mystcb/blog:ref:refs/heads/main"]
    }

    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }
  }

}

Now we have our two policy documents as Terraform data objects, we can then pull them together to create the role

Creation of the IAM Role

For us to create the role, we need to pull together all the bits we have created so far. This will mean doing a few special tricks with the data resources.

Firstly we need to create an IAM Policy resource, that the IAM Role resource can use to attach to the newly created role.

The IAM Policy resource require 3 elements, the name, path , and the policy in JSON format. While the name and path can be as custom as you need, the policy in JSON format is what might trip a few people up. The IAM Policy Document data object has just one Attribute output tha can be referenced here: json. This conversion means we can quickly add the three together to make the IAM Policy object in AWS.

resource "aws_iam_policy" "website_colins_blog" {
  name   = "access_to_website_colins_blog_s3"    # This is my example name!
  path   = "/"                                   # Root path for ease
  policy = data.aws_iam_policy_document.website_colins_blog_policy.json
}

Next, we need to create the role itself which has two key elements, the name and the assume_role_policy. The name is as you want this to be however, the assume_role_policy will be needed to let AWS IAM know what can assume this role. In our case, it is the JSON output from our second IAM Policy Document.

resource "aws_iam_role" "website_colins_blog_github_role" {
  name               = "access_to_website_colins_blog_s3_role"      # This is my example name!
  assume_role_policy = data.aws_iam_policy_document.website_colins_blog_role_assumption.json
}

Great! Now we have role, and we can assume it it - well not exactly, one last step. With IAM, you can attach multiple policies to a single role, which if you created in line with the aws_iam_role resource it can make it a little more complicated, and very long. The AWS provider allows us two methods to manage the role’s policies. One is through managed_policy_arns/inline_policy or aws_iam_policy_attachment. The former two work very much the same, but takes exclusive authority over the state of the IAM Role itself. This means if you attach policies using the latter resource object, you will find Terraform getting stuck in a cycle. For this example, I am using the policy attachment resource.

resource "aws_iam_role_policy_attachment" "website_colins_blog" {
  role       = aws_iam_role.website_colins_blog_github_role.name
  policy_arn = aws_iam_policy.website_colins_blog.arn
}

This block is where all the references from before make this easier. This is where the role resource object, and the policy resource object come together to create the role. Now, AWS is aware of GitHub, its OpenID Connect provider, and we have given a specific repo’s GitHub actions that run on the main branch access to assume a role in AWS, which give it access to AWS S3. The role assumption will use temporary credentials for each of the runs!

One last bit, this part will enable you to get the ARN for the role, which will be required for the configuration of GitHub.

output "website_colins_blog_role_arn" {
  description = "The role ARN for the Website: Colin's Blog Role"
  value       = aws_iam_role.website_colins_blog_github_role.arn
}

Very simple, it just outputs the ARN for the role which will need to be copied to GitHub. For the rest of the blog, I am going to be using an example role in my examples. This will not work, so make sure you are getting your role that matches your account. The ARN we will be using will be

arn:aws:iam::12326264843:role/access_to_website_colins_blog_s3_role

One IAM Role created, with federation to GitHub

GitHub Actions/Workflow Updates

To enable this, we need to update the workflow yaml file, and this is probably the easiest bit of the whole post!

Add the Role ARN as a secret

This is where we need to move back to GitHub and grab the ARN from above, and add this as an add the URL to the Repository Secrets. You should be able to find your version at https://github.com/<yourname/org>/<yourrepo>/settings/secrets/actions. This is under Settings -> Secrets and variables -> Actions and click New Secret. Enter in a name for the secret, and its value which I have put in:

Name: AWS_ROLE_ARN
Secret: arn:aws:iam::12326264843:role/access_to_website_colins_blog_s3_role

Entering in the new secret with the role ARN

Once you have added that, make sure to remove the two existing Repository Secrets, in my case I called them

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

The AWS CLI will always use the keys over role assumption in it’s priority so always best to remove them. With the two older secrets removed you should now have just the AWS_ROLE_ARN and AWS_DEFAULT_REGION

Only the final two secrets left

Update the Workflow YAML file

For GitHub actions to be able to assume the role, there are two changes that need to be made to the workflow yaml file. The first one, will be the need to enable the workflow to interact with GitHub’s OIDC Token endpoint. Part of the assumption process will require us to identify as a web identity from GitHub to have AWS know who we are. As such you will need to add additional permissions to the job. Specifically the following

id-token to write
contents to read

So it should look something like this:

jobs:
  hugo_build_and_deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

Later on in the pipeline, where we configured the AWS credentials before, you will need to remove the older secret variables, and put the new secrets in:

- name: Configure AWS Credentials
  id: aws-credentials-configure
  uses: aws-actions/configure-aws-credentials@v1
  with:
    role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
    aws-region: ${{ secrets.AWS_DEFAULT_REGION }}

To see this whole file in context, I would recommend having a look at this blog’s workflow on GitHub!

All you need to do now, is run the GitHub Actions workflow, and make sure it works! With my blog workflow, it was pretty much a drop in replacement for the IAM credentials. There were a few minor issues with my workflow, but nothing that following this wouldn’t have resolved fo me!

Some minor issues

As you can see, it wasn’t exactly first time running for me! It did take a while, and also I had placed a --acl public-read as part of the aws s3 sync command, which the new bucket I created had been set to block public ACLs!

Just a few mistakes!

There was one other issue, and that was with the GITHUB_TOKEN that is used. In normal operation, without the added permissions, this token worked fine with an additional Marketplace action called GitHub Deployments. However, changing this over to allow the OpenID Connect feature, it meant that the token mysteriously stopped working.

Switching to use a Fine-grained PAT

On the 18th of October 2022, GitHub offered up a new service called Fine-grained personal access tokens. The idea is that rather than creating a very open Personal Access Token (PAT), you could create a token that was very limited in it’s reach. It is still in beta as I write this blog (28th Feb 2023).

Using this beta feature, I was able to create a new token, limiting it to specifically the blog repo, and specific permissions. The screenshot below shows the details about the new PAT. (I am aware I could probably reduce the permissions a little more!)

A new Fine-grained Personal Access Token (beta)

From here, I added a new respository secret called REPO_TOKEN with the value of the newly generated token, and then updated the part in the workflow that needed it:

- name: Set Deploy Status
  id: deploy-status-start
  uses: bobheadxi/deployments@v1
  with:
    step: start
    token: ${{ secrets.REPO_TOKEN }}
    env: Prod
    ref: ${{ github.ref }}

Round up

Hopefully what I have shown you in this post is how to move away from IAM Credentials, and use the OpenID Connect features of both AWS and GitHub to enable role based assumption to gain access to an S3 bucket that stores, in my case, a static website.

If you do have any questions, comments, please let me know!