How to apply auto tags to new resources in AWS using Cloud Custodian

Cloud Custodian logo — cloudcustodian.io | AWS logo — aws.amazon.com

Following on the previous post about enforcing tags with Cloud Custodian.

Simple AWS tag enforcement with Cloud Custodian

I thought it would be nice to show how to implement auto-tagging of new AWS resources using the same set of tools. This is a great way to standardize tagging across the board; However, it may be a good idea to think twice before implementing such approach.

Without further delay, here are the steps to start auto tagging resources in your AWS account.

We’ll go ahead and keep our IaC approach and either use CloudFormation or Terraform to the template. For this post, I’ll use CloudFormation.

IAM Role and Instance Profile

As stated on my previous posts: “you always start with IAM” and since we are using an instance to…

1. Create the Lambdas and CloudWatch Events.
2. Pass the appropriate roles to the new resources
3. List, Fetch, and Put objects to S3.

…we’ll need to either get very granular (as we should) and add individual Statements or policies to control the access of this particular instance OR you can take the lazy way out… I mean the more productive way! and use managed policies to create the role. Please keep in mind that this is the same role that will be assumed by the Lambda to make changes to the new resources

Once you create the role, make sure you include sts policy to allow lambda services to assume the role.

NOTE: An alternative is to include another role just for Custodian Lambdas and limit the Ec2 bound role to the bare minimum.

S3 Bucket (Cloud Custodian Files)

This S3 Bucket will host our Cloud Custodian policies and logs produced from each custodian run — keep in mind that those logs can also be pushed to CloudWatch, but that’s different post.

Make sure to add a bucket policy with full rights where the Custodian role is the only principal.

The aim here, is to be able to create a script in our deployment, to pull those files on a scheduled basis AND we also want to make sure we create a script to upload the logs on a regular basis — unless you want to use CloudWatch for the logging portion.

CloudTrail + S3 Bucket (Trail Logs)

This one is pretty self explanatory, we need a CloudTrail (Trail) to be enabled in order to trigger the CloudWatch Event which will then execute the Custodian Lambda to auto-tag the new resources.

Make sure you include the appropriate bucket policies for S3. eg:

Statement:
-
Sid: "AWSCloudTrailAclCheck"
Effect: "Allow"
Principal:
Service: "cloudtrail.amazonaws.com"
Action: "s3:GetBucketAcl"
Resource:
!Sub |-
arn:aws:s3:::${BucketName}
-
Sid: "AWSCloudTrailWrite"
Effect: "Allow"
Principal:
Service: "cloudtrail.amazonaws.com"
Action: "s3:PutObject"
Resource:
!Sub |-
arn:aws:s3:::${BucketName}/AWSLogs/${AWS::AccountId}/*
Condition:
StringEquals:
s3:x-amz-acl: "bucket-owner-full-control"

EC2 Instance (Cloud Custodian)

Cloud Custodian is a very light weight tool and therefore we don’t need high specs for this instance. Feel free to use the free tier type and storage class.

For the Userdata script, we’ll include the following:

#!/bin/bash -xe
sudo yum update -y
sudo yum install python3 python3-pip -y
sudo pip3 install c7n
sudo mkdir /opt/custodian
echo $'#!/bin/bash\nBUCKET=${BucketName}\nsudo aws s3 cp s3://$BUCKET/ /opt/custodian/ --recursive' > /home/ec2-user/custodiansync
sudo chmod 775 /home/ec2-user/custodiansync
sudo mv /home/ec2-user/custodiansync /etc/cron.hourly/custodiansync sudo chmod +x /etc/cron.hourly/custodiansync

Please note that we are using the substitution function in order to pass the custodian bucket parameter to the new script we are creating locally. This script should run on an hourly basis in order to pull the logs and policies from S3.

Cloud Custodian Policy and Execution

For this particular post, we’ll focus on just EC2 instances. We want to make sure that all instances are auto-tagged with the username of whoever created it. With that in mind, we’ll go ahead and create the following yaml file:

policies:
- name: ec2-auto-tag-user
resource: ec2
mode:
type: cloudtrail
role: arn:aws:iam::{account_id}:role/Custodian
events:
- RunInstances
timeout:
- 10
filters:
- tag:Creator: absent
actions:
- type: auto-tag-user
tag: Creator
principal_id_tag: CreatorId

NOTE: Make sure you are using the name you gave to the role you created at the beginning of this post!

Save the file and upload to the Custodian S3 Bucket we created a while back. The files will eventually be downloaded… in about an hour…

Alright, now with the policy file in our instance, we can manually run the policy by using the following command:

sudo /usr/local/bin/custodian run --output-dir=./logs --cache-period 0 --region us-east-1 /opt/custodian/policies/ec2-auto-tag-user.yml

Make sure you adjust the above to match your aws resource region, file name, and the path of the policy. The successful execution of the policy will create a Lambda Function and a CloudWatch Event Rule based on CloudTrail events matching “RunInstances”; Meaning that any new instances will trigger the lambda to add the needed tags. ALL DONE! you can now stop the instance…

Conclusion

Hopefully the above gives you a better idea on how to better implement Cloud Custodian, to your company or clients cloud infrastructure, in a way that improves cloud governance and makes your architecture more responsive and receptive. Please note that the above is only a small demonstration of Cloud Custodian capabilities! Amazing uh?

BONUS: since the policy runs on CloudTrail Events, you can add more policies to the same document using the appropriate resource you wish to monitor and the respective trail event. eg:

VPC - CreateNatGateway
VPC - CreateKeyPair
VPC - CreateNetworkAcl
VPC - CreateNetworkInterface
VPC - CreateRouteTable
VPC - CreateVpc
VPC - CreateVPCEndpoint
VPC - CreateVpcPeeringConnection
VPC - CreateVpnConnection
VPC - CreateVpnGateway
VPC - CreateSecurityGroup
EFS - CreateFilteSystem

If you like this kind of posts, don’t forget to subscribe to my weekly newsletter!

SimpleCloudBits