Introduction

I was excited to tackle today’s topic of using a CI/CD tool, GitHub Actions, to automate part of the deployment of my Notes app. One of my favorite aspects of working with technology is getting to automate things. Much of what I built in my last post was done manually in a graphical interface, which wasn’t as exciting for me, but was a necessary step in the learning process. So, let’s see how I automated part of the deployment process using GitHub Actions.

This post is part of a series where I create a basic CRUD app and improve upon the development and deployment of the app using DevOps tools and techniques.

Title
Creating a Notes App with FARM Stack
Containerize a FastAPI App with Docker
Deploying a FastAPI Container to AWS ECS
Setting Up GitHub Actions to Deploy to ECS
Using Terraform for ECS/EC2

Setup

Create workflow in GitHub Actions

After looking at some guides, I started by going to the repo for my Notes app and going to the Actions tab, then New Workflow. From there I searched for “ECS” and selected the one result - Deploy to Amazon ECS.

I think this was a good place to start, but I realized halfway through that this template was a bit out of date. I ended up having to pull pieces from other specific workflows to update most sections. This Github Docs page along with the official AWS Actions helped me figure out all the modifications I ended up needing to make.

Create JSON task definition for ECS

The next thing I needed to do was to get the JSON version of the task definition that I created in my last post. I did this by going to ECS in the AWS console, then Task definitions, selecting my task - notes-api-task, selecting the latest revision, and selecting the JSON tab. I now know another way to do this via the AWS CLI:

aws ecs list-task-definitions

Which will give you the ARN to use the following command to get the JSON:

aws ecs describe-task-definition --task-definition arn:aws:ecs:us-west-2:<AWS ID>:task-definition/notes-api-task

So I copied the JSON and pasted into a file in the root of my notes project folder and called it notes-api-task.json. I also found out later that I needed to modify this task definition.

Add service to ECS cluster

Next, I needed to stop the existing task running in my ECS cluster and change it to a service. I stopped the existing task by going to the ECS console, then my notes-api cluster, then the Tasks tab, selected the running task, and hit Stop.

To create the service, I went to Services tab in my ECS cluster, clicked Create, and changed the following:

Launch type: EC2
Task definition - Family: notes-api-task (created previously)
Service name: notes-api-service

With the task definition and service in place, I went back to the GitHub Actions workflow and edited the environment variables it had:

env:
  AWS_REGION: us-west-2                   # set this to your preferred AWS region, e.g. us-west-1
  ECR_REPOSITORY: notes-api           # set this to your Amazon ECR repository name
  ECS_SERVICE: notes-api-service                 # set this to your Amazon ECS service name
  ECS_CLUSTER: notes-api                 # set this to your Amazon ECS cluster name
  ECS_TASK_DEFINITION: notes-api-task.json # set this to the path to your Amazon ECS task definition
                                               # file, e.g. .aws/task-definition.json
  CONTAINER_NAME: notes-api-container           # set this to the name of the container in the
                                               # containerDefinitions section of your task definition

It looked like I had everything in place, so I made a new commit to the main branch to kick off the workflow and it errored on the first real step - AWS credentials.

screenshot

It seemed obvious as I saw it and from there I went down a bit of a rabbit hole of methods and best practices for authorizing GitHub to talk to AWS.

Connect GitHub to AWS

Add identity provider to AWS

After some reading and consideration, I decided to use the OpenID Connect (OIDC) method for authentication as it seemed to be the best practice as well as being simple enough to learn in one sitting. I started by going to the IAM console in AWS, then Indentity providers, and clicked Add provider.

I then entered the following:

  • Provider type: OpenID Connect
  • Provider URL: token.actions.githubusercontent.com
  • Audience sts.amazonaws.com

Create role for GitHub Actions

Then I added a role for GitHub Actions by going to Roles within IAM, then Create role. I selected Custom trust policy and added the following JSON:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": "sts:AssumeRoleWithWebIdentity",
			"Principal": {
				"Federated": "arn:aws:iam::<AWS ID>:oidc-provider/token.actions.githubusercontent.com"
			},
			"Condition": {
				"StringLike": {
					"token.actions.githubusercontent.com:sub": "repo:rickthackeray/notes:*"
				},
				"StringEquals": {
					"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
				}
			}
		}
	]
}

With that, I pushed a commit to kick off a new instance of the workflow and saw success! Well, I got a check mark for the first section: “Configure AWS credentials” - it failed on the very next one: “Login to Amazon ECR”.

Add permission to authenticate with ECR

To remedy this, I went to the IAM console, then Roles, and selected my “github-actions” role. From there I went to Permissions, Add Permissions, and Create Inline Policy. I then selected the JSON tab and added this:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "GetAuthorizationToken",
			"Effect": "Allow",
			"Action": [
				"ecr:GetAuthorizationToken"
			],
			"Resource": "*"
		}
	]
}

I then pushed a new commit to rerun the workflow and got a second check mark! Of course it failed on the step after that.

screenshot

Build, tag, and push Docker image to ECR

I was at 2/5 for successful steps in my workflow. Up to this point I didn’t need to modify the workflow template other than inserting the initial configuration settings in the form of environment variables. That was about to change.

The error on this step was that it couldn’t find the Dockerfile. I realized that my folder structure was the culprit. See, the Dockerfile and the rest of the code for this part of the app was in a subfolder: “backend”. I fixed this by changing the run command, adding cd backend. The final version of the step looked like this:

    - name: Build, tag, and push docker image to Amazon ECR
      env:
        REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        REPOSITORY: ${{ env.ECR_REPOSITORY }}
        IMAGE_TAG: ${{ env.IMAGE_TAG }}
      run: |
        cd backend
        docker build -t $REGISTRY/$REPOSITORY:$IMAGE_TAG .
        docker push $REGISTRY/$REPOSITORY:$IMAGE_TAG

I suspect there are better ways to do this and I look forward to learning that one day.

I could see in the Workflow log that the image was successfully building, but there was still an error on this step that indicated my “github-actions” role didn’t have the right permissions. I eventually fixed it by adding the following inline policy to the role:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AllowPushPull",
			"Effect": "Allow",
			"Action": [
				"ecr:BatchGetImage",
				"ecr:BatchCheckLayerAvailability",
				"ecr:CompleteLayerUpload",
				"ecr:GetDownloadUrlForLayer",
				"ecr:InitiateLayerUpload",
				"ecr:PutImage",
				"ecr:UploadLayerPart"
			],
			"Resource": "arn:aws:ecr:us-west-2:<AWS ID>:repository/*"
		}
	]
}

With that in place, I got another check mark - 3/5!

screenshot

And the next step failed, unsurprisingly. I was getting an error that it couldn’t update the task definition with the new image.

Configure workflow to update ECS task definition

One of the problems seemed to be related to the filename format for the image. The workflow was using the hash of the commit to tag the new build. So, I tried setting it to a static name of “Latest” and that worked. I’m sure there was a way to get the dynamic tagging convention to work, but it wasn’t necessary at this point.

This didn’t get me the next check mark, though. The new error sounded like a permissions issue. I struggled with finding the right specific permissions for ECS that the workflow needed. So, I ended up opening it up fully for ECS. This is not ideal in an environment where security is important as it does not meet the principle of least privilege, but it allowed me to continue learning in my fairly low risk environment. Here is the policy that I added to my github-actions role:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"ecs:*"
			],
			"Resource": "*"
		}
	]
}

After that, I pushed a new commit and got another check mark - 4/5! It was time to get that last one!

Configure workflow to deploy ECS task

This one was possibly the most difficult. The workflow logs weren’t really giving an error - just showing that the attempt at deploying the ECS task was timing out. It took a lot of troubleshooting effort to find out that the issue was with how I was handling secrets that the app needed.

You can see in a previous post that I used the volume feature of Docker to add a keys.env file at image runtime. The problem was that the EC2 instance that ECS was controlling didn’t have this file and the task definition didn’t have the volume parameter for the docker run command configured.

To get the keys.env file onto the EC2 instance, I simply used scp to copy it over. I’m guessing this isn’t how this is typically done, as I can see issues coming up when working with a team or at larger scales, but again it allowed me to continue learning more instead of fixating on something with lower return. The phrase “Don’t let perfect be the enemy of good” comes to mind.

Anyway, I could tell by ssh’ing into the EC2 instance and manually running the container that this worked, but I still needed to update the task definition to include the file as a Docker volume at runtime. I did that by modifying the notes-api-task.json file I created before, adding mountPoints and volumes sections. Here is what the file ended up as:

{
    "taskDefinitionArn": "arn:aws:ecs:us-west-2:<AWS ID>:task-definition/notes-api-task:9",
    "containerDefinitions": [
        {
            "name": "notes-api-container",
            "image": "<AWS ID>.dkr.ecr.us-west-2.amazonaws.com/notes-api",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "notes-api-container-8000-tcp",
                    "containerPort": 8000,
                    "hostPort": 80,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [],
            "mountPoints": [
                {
                    "sourceVolume": "keys",
                    "containerPath": "/app/keys.env",
                    "readOnly": false
                }
            ],
            "volumesFrom": []
        }
    ],
    "family": "notes-api-task",
    "revision": 9,
    "volumes": [
        {
            "name": "keys",
            "host": {
                "sourcePath": "/home/ec2-user/keys.env"
            }
        }
    ],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2"
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "1024",
    "memory": "100",
    "registeredAt": "2023-11-15T01:19:16.910Z",
    "registeredBy": "arn:aws:iam::<AWS ID>:user/<AWS USER>",
    "tags": []
}

One more commit later and I had 5/5 check marks!

screenshot

Conclusion

I successfully set up my first CI/CD pipeline with Github Actions. I can now work on the backend of my notes app and easily deploy new versions to production by simply pushing a new commit to the main branch.

I really enjoyed this one. It was so gratifying to put in a lot of effort and to see the results of that effort in the form of check marks when each step succeeded. There were certainly times I entered a kind of flow state working on this where the rest of the world dissolved away. I look forward to learning more and putting in the kind of effort that can lead to such a lovely state.

You can see the Notes app in action here: http://notes.rickt.io/

Cheers!
Rick

 

 

Resources