Using Boto3 Paginator

Using Boto3 Paginator

If you are retrieving any data from AWS using Boto3, you should make a habit of using Paginator for the day when your data-set gets too large for a single call to handle.

Here's a simple illustration on how to lookup paginator arguments and returns and how to use it in your code.

Go to Boto3 Docs
Every service has a paginators. Write your code based on the class definition as shown above.
Here's the code:

thisClient = boto3.client("iam", aws_access_key_id=credentials['AccessKeyId'],
                                 aws_secret_access_key=credentials['SecretAccessKey'],
                                 aws_session_token=credentials['SessionToken'])
paginator = thisClient.get_paginator('list_users')
    response_iterator = paginator.paginate()
    for response in response_iterator:
        for key in response.get('Users'):
            thisUserId = key.get('UserId')
            thisUserName = key.get('UserName')

Managing Terraform State file

Terraform Notes

Moving resources and backing up tf state

This will illustrate process of migrating resources from one state to another as well as backing up and reverting to different states. 

In your working directory, you should have at least terraform.tfstate. If you have defined an external backend, then this file may be cached under .terraform directory (and your actual changes are happening on remote location). 

if your main.tf is this:

variable "timestamps" {
  type    = bool
  default = false
}

resource "null_resource" "exampleA" {
  triggers = {
    key = formatdate("YYYY-MM-DD hh:mm:ssZZZZZ", timestamp())
  }
}

Then your intial terraform.tfstate would look like this:

{
  "version": 4,
  "terraform_version": "0.12.24",
  "serial": 1,
  "lineage": "d0427b0f-6517-25bb-f708-6666666666",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "null_resource",
      "name": "exampleA",
      "provider": "provider.null",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "8139684392621207633",
            "triggers": {
              "key": "2020-04-23 17:41:12+00:00"
            }
          },
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

To get list of all your resource, you can run "terraform state list"

From the output from above, you can use the mv command to move resources to be managed by different terraform state. You can also use this command to refactor the name of the resource.

terraform state mv 

-backup='terraform.backup.tfstate' 

-state-out='terraform.new.tfstate' 

 null_resource.exampleA 

 null_resource.exampl

The above command will create two additional terraform state files.

terraform.tfstate

{
  "version": 4,
  "terraform_version": "0.12.24",
  "serial": 2,
  "lineage": "d0427b0f-6517-25bb-f708-6666666666",
  "outputs": {},
  "resources": []
}

terraform.new.tfstate

{
  "version": 4,
  "terraform_version": "0.12.24",
  "serial": 1,
  "lineage": "c7e1be5e-5363-39be-7078-6666666666",  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "null_resource",
      "name": "exampleB",
      "provider": "provider.null",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "8139684392621207633",
            "triggers": {
              "key": "2020-04-23 17:41:12+00:00"
            }
          },
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

terraform.backup.tfstate

{
  "version": 4,
  "terraform_version": "0.12.24",
  "serial": 1,
  "lineage": "d0427b0f-6517-25bb-f708-6666666666",  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "null_resource",
      "name": "exampleA",
      "provider": "provider.null",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "8139684392621207633",
            "triggers": {
              "key": "2020-04-23 17:41:12+00:00"
            }
          },
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

Note that lineage of terraform.tfstate and terraform.backup.tfstate is going to be identical.

If you run terraform state list now you'll get nothing.

To go back to previous tf state, you can run the command in reverse where source (-state) is the new terraform state file and destination (-state-out) is the original state file.

terraform state mv 
-backup='terraform.backup2.tfstate' 
-state-out='terraform.tfstate' 
-state='terraform.new.tfstate'
 null_resource.exampleB 
 null_resource.exampleA

If this is just a local state, you can also just rename terraform.backup.tfstate to terraform.tfstate.

Now if you run terraform state list again you'll get your resource back.

Terraform for_each caution

Terraform Notes

For_Each Loop Caution

When using for_each, be sure to be mindful of what you want these new resources to be indexed as. Because if you ever try to change the index name, the resources must be destroyed and recreated. 

Example map variable

variable "account_map"{
    default = {
        "TEST-A-Key" = {
            "name" = "TEST-A-Name"
            "email" = "TEST-A@me.com"
        },
        "TEST-B-Key" = {
            "name" = "TEST-B-Name"
            "email" = "TEST-B@me.com"
        }
    }
}

When you create a resource based on the above map, you get this:

resourceType.name["TEST-A-Key"]
resourceType.name["TEST-B-Key"]

Unfortunately, in the above example (and in my real world case), I've tied the index name to be same as the name of the account. So when I went to updated the account name (the above variable is created from a file), it attempted to delete all my account resources. So now my only choice is to delete state of these resources and re-import OR live with mis-matched account name and index string of the resources.

So take caution and use some index string that describes the variable, not the specific content. Or maybe you don't mind destroy and create whenever you want to change a name. 

AWS Custom Config Rule from Org

Deploy AWS Config Rule from Org

If you have AWS Org configured, you can deploy Config Rule from a single location out to all the accounts in the same Org. You can do this for both AWS Managed and your Custom Rules. 

References:

AWS Dev Guide: config-rule-multi-account-deployment

Details:

• Max 150 rules
• All necessary permissions and roles must be pre-configured

Advantages:

• Deploy from central location
• Does not allow accounts from editing the rules

Setup

Setup Root Account

• Configure Aggregator
• Configure S3 Logging Bucket for Config* 
• Configure SNS Topic*
• Configure Lambda Role
• Your usual Lambda Role permissions
• Ability to Assume Role to the Recorder Role name in all accounts (this can be extracted from "executionRoleArn" of the Event object)
• Configure Lambda Function
• Need to allow this function to be executed by all accounts in this Org

Setup All Accounts

• Configure Recorder Role 
• use same name across all accounts
• need necessary permission to 
• write to Config 
• write to S3 Logging Bucket in Root Account*
• write to SNS Topic in Root Account*
• Trust Lambda Role in Root Account to AssumeRole
• and read/edit necessary resources for it to do what you want
• Turn on Recording

*The S3 and SNS does not have to be in Root Account, just put it somewhere common

Execution

AWS Doc API PutOrganizationConfigRule

AWS Custom Config Cross-Account

How to AWS Custom Config Cross-Account

Lambda - Python

Account 999999999

• This is where you'll host the Lambda code

Account 888888888

• This is where you'll host the Config Rule

Account 999999999

Follow the previous instruction for the Python code, update the lambda_handler with following:

def lambda_handler(event, context):
    evaluations = []
    test_mode = False
    thisRegion = 'us-east-1'
    print('Event: ',event)
    invoking_event = json.loads(event["invokingEvent"])
    print('Invoking Event: ',invoking_event)
    notification_time = invoking_event["notificationCreationTime"]
    print('Invoked Time: ', notification_time)
    result_token = event["resultToken"]
    print('Result Token: ', result_token)
    # In our test event, we have defined the result_token to be XYZ
    if result_token == 'XYZ':
        test_mode = True
        
    # this is passed to us from Config Rule, if this exists, then Assume Role
    ruleParameters = json.loads(event["ruleParameters"])
    if 'executionRole' in ruleParameters.keys():
        print('Assume Role')
        executionRole = ruleParameters["executionRole"]
        print(executionRole)
        sts_client = boto3.client('sts')
        assume_role_response = sts_client.assume_role(RoleArn=executionRole, RoleSessionName="configLambdaExecution")
        credentials = assume_role_response['Credentials']
        config = boto3.client("config", region_name=thisRegion,
                        aws_access_key_id=credentials['AccessKeyId'],
                        aws_secret_access_key=credentials['SecretAccessKey'],
                        aws_session_token=credentials['SessionToken']
                       )
        
    else:
        credentials = []
        config = boto3.client("config")
        print('Self Run')
    # pass in credential and time stamp and get back eval result
    evaluations = evaluate_compliance(notification_time, credentials)
    print(evaluations)
    print(result_token)
    result = config.put_evaluations(
            Evaluations = evaluations,
            ResultToken = result_token,
            TestMode = test_mode
            )
    
    metaData = result["ResponseMetadata"]
    statusCode = metaData["HTTPStatusCode"]
    return {
        'statusCode': statusCode,
        'body': json.dumps(result)
    }

Be sure your execution role for this function has at least the following. The first statement allows the function to send logs to cloudwatch. The second statement allows this function to assume the role of Config_Role from another account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:us-east-1:999999999:log-group:/aws/lambda/config_test:*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:us-east-1:999999999:*"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::888888888:role/config_role
        }
    ]
}

Add a Resource-based policy to your Lambda function so that it can be used by another account. This can only be done via CLI or API call.
Run this CLI command using credential of account 99999999

aws lambda add-permission 
  --function-name config_test
  --region us-east-1 
  --statement-id 1001 
  --action "lambda:InvokeFunction" 
  --principal config.amazonaws.com 
  --source-account 888888888 

Account 888888888

Create a new role in this account with at least following permission. Be sure to have specific permission so that you can evaluate your resources.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "config:PutEvaluations"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

The new role also needs trust relationship to ALLOW it be assumed by the Lambda function

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::999999999:role/service-role/lambdaConfigRole"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

Now create the new Config Rule
Provide the Lambda function that you created in account 99999999:

Set your trigger type and resource type(s).

Fill out the Rule parameter with the role that you want the Lambda function to use:

Done. Now try running the config rule manually by clicking on the blue Re-evaluate button in account 8888888888 and watch the Cloudwatch logs in account 999999999.

AWS Config Custom Rule

How to AWS Custom Config 

Lambda - Python

Want to know how to setup your lambda Python function for AWS Custom Config? Here's what you need to know to get started. 

Here's a really simplified version of the Python code that you need to get started. You should add some try and catch errors and parse whatever input you are looking for.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

def lambda_handler(event, context): # you can check what invoked it invoking_event = json.loads(event["invokingEvent"]) # you only need this for resource change trigger configuration_item = invoking_event["configurationItem"] # you can check what rule parameters were passed in rule_parameters = json.loads(event["ruleParameters"]) compliant, annotation = evaluate_compliance(some_argument) config = boto3.client("config") # this example assumes resource trigger, for time trigger, you can have the above function spit out a list of resources config.put_evaluations( Evaluations=[ { "ComplianceResourceType": configuration_item["resourceType"], "ComplianceResourceId": configuration_item["resourceId"], "ComplianceType": compliant, "Annotation": annotation, "OrderingTimestamp": configuration_item["configurationItemCaptureTime"] }, ], ResultToken=result_token, ) def evaluate_compliance(argument): compliant = "COMPLIANT" annotations = [] if blah: compliant = "COMPLIANT" annotations.append("blah") else: compliant = "NON_COMPLIANT" annotations.append("different blah") return compliant, " ".join(annotations)

To understand how to fill in the logic of your code, you can look at what you get in the event payload.

Inputs

Sample scheduled trigger:

1 2 3 4 5 6 7 8 9 10 11 12

{ "version":"1.0", "invokingEvent":"{\"awsAccountId\":\"999999999\",\"notificationCreationTime\":\"2020-03-23T02:42:51.511Z\",\"messageType\":\"ScheduledNotification\",\"recordVersion\":\"1.0\"}", "ruleParameters":"{\"executionRole\":\"arn:aws:iam::9999:role/blah\"}", "resultToken":"XYZ", "eventLeftScope":false, "executionRoleArn":"arn:aws:iam::9999999:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig", "configRuleArn":"arn:aws:config:us-east-2:999999999:config-rule/config-rule-kfabou", "configRuleName":"my-test-rule1", "configRuleId":"config-rule-kfabou", "accountId":"9999999999" }

Above's invoking event:

1 2 3 4 5 6

{ "awsAccountId":"999999999", "notificationCreationTime":"2020-03-23T02:42:51.511Z", "messageType":"ScheduledNotification", "recordVersion":"1.0" }

Rule parameter is the values you provide in Key:Value pair under your Config rule setup

Sample resource change trigger:

1 2 3 4 5 6 7 8 9 10 11 12

{ "version":"1.0", "invokingEvent":"{\"configurationItemDiff\":null,\"configurationItem\":{\"relatedEvents\":[],\"relationships\":[{\"resourceId\":\"ANPASP666ZVAXGDCE5SDN\",\"resourceName\":\"Read_Only_EC2\",\"resourceType\":\"AWS::IAM::Policy\",\"name\":\"Is attached to CustomerManagedPolicy\"}],\"configuration\":{\"path\":\"/\",\"userName\":\"FoxySugar\",\"userId\":\"AIDASP666ZVA36XQGO6DD\",\"arn\":\"arn:aws:iam::9999999999:user/FoxySugar\",\"createDate\":\"2020-01-07T14:06:13.000Z\",\"userPolicyList\":[],\"groupList\":[],\"attachedManagedPolicies\":[{\"policyName\":\"Read_Only_EC2\",\"policyArn\":\"arn:aws:iam::9999999:policy/Read_Only_EC2\"}],\"permissionsBoundary\":null,\"tags\":[]},\"supplementaryConfiguration\":{},\"tags\":{},\"configurationItemVersion\":\"1.3\",\"configurationItemCaptureTime\":\"2020-03-23T02:02:28.440Z\",\"configurationStateId\":1584928948999,\"awsAccountId\":\"999999999\",\"configurationItemStatus\":\"ResourceDiscovered\",\"resourceType\":\"AWS::IAM::User\",\"resourceId\":\"AIDASP666ZVA36XQGO6DD\",\"resourceName\":\"FoxySugar\",\"ARN\":\"arn:aws:iam::9999999:user/FoxySugar\",\"awsRegion\":\"global\",\"availabilityZone\":\"Not Applicable\",\"configurationStateMd5Hash\":\"\",\"resourceCreationTime\":\"2020-01-07T14:06:13.000Z\"},\"notificationCreationTime\":\"2020-03-23T02:40:01.999Z\",\"messageType\":\"ConfigurationItemChangeNotification\",\"recordVersion\":\"1.3\"}", "ruleParameters":"{}", "resultToken":"XYZ==", "eventLeftScope":false, "executionRoleArn":"arn:aws:iam::99999999:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig", "configRuleArn":"arn:aws:config:us-east-2:999999999:config-rule/config-rule-kfabou", "configRuleName":"my-config-rule1", "configRuleId":"config-rule-kfabou", "accountId":"99999999999999" }

Above's invoking event:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73

{ "configurationItemDiff":null, "configurationItem":{ "relatedEvents":[ ], "relationships":[ { "resourceId":"ANPASP666ZVAXGDCE5SDN", "resourceName":"Read_Only_EC2", "resourceType":"AWS::IAM::Policy", "name":"Is attached to CustomerManagedPolicy" } ], "configuration":{ "path":"/", "userName":"FoxySugar", "userId":"AIDASP666ZVA36XQGO6DD", "arn":"arn:aws:iam::9999999999:user/FoxySugar", "createDate":"2020-01-07T14:06:13.000Z", "userPolicyList":[ ], "groupList":[ ], "attachedManagedPolicies":[ { "policyName":"Read_Only_EC2", "policyArn":"arn:aws:iam::9999999:policy/Read_Only_EC2" } ], "permissionsBoundary":null, "tags":[ ] }, "supplementaryConfiguration":{ }, "tags":{ }, "configurationItemVersion":"1.3", "configurationItemCaptureTime":"2020-03-23T02:02:28.440Z", "configurationStateId":1584928948999, "awsAccountId":"999999999", "configurationItemStatus":"ResourceDiscovered", "resourceType":"AWS::IAM::User", "resourceId":"AIDASP666ZVA36XQGO6DD", "resourceName":"FoxySugar", "ARN":"arn:aws:iam::9999999:user/FoxySugar", "awsRegion":"global", "availabilityZone":"Not Applicable", "configurationStateMd5Hash":"", "resourceCreationTime":"2020-01-07T14:06:13.000Z" }, "notificationCreationTime":"2020-03-23T02:40:01.999Z", "messageType":"ConfigurationItemChangeNotification", "recordVersion":"1.3" }

Output:

1 2 3 4 5 6 7 8 9 10 11 12 13

{ "Evaluations": [ { "Annotation": "string", "ComplianceResourceId": "string", "ComplianceResourceType": "string", "ComplianceType": "string", "OrderingTimestamp": number } ], "ResultToken": "string", "TestMode": boolean }

This is from this documentation.

Explanation (Lines):
2: This is array of compliance report. You can include as many as you like.
4. This is optional description
5. This is the ID of the resource
6. This is the resource type, this must be one supported by Config. Click here.
7. This must be one of following:  

• COMPLIANT
• NON_COMPLIANT
• NOT_APPLICABLE
• INSUFFICIENT_DATA

8. This is timestamp
11. This must be the same ResultToken that function received from Config.
12. If this is set to true, then nothing is actually reported back to Config.  


With this in mind, you should be able to make sense of these sample AWS Config Lambda codes such as one found here.

Terraform Cloud current workspace name

Terraform Notes

How to get current workspace name

Working in TF Cloud and you want your tf file to pull your current workspace name. These are what I would call TF Cloud environment variables or global variables or backend data or stuff you can self reference. 

Here is my main.tf under my Github repo that is tied to my Terraform Cloud account. 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

variable "ATLAS_WORKSPACE_NAME"{ } locals{ version = "1.1" } resource "null_resource" "example" { triggers = { key = "hello" value = formatdate("YYYY-MM-DD hh:mm:ssZZZZZ", timestamp()) } } output "version"{ value = local.version } output "workspace"{ value = var.ATLAS_WORKSPACE_NAME }

The most important part is lines 1-2. Even though I would consider "ATLAS_WORKSPACE_NAME" to be global variable in a traditional sense, Terraform creates a dependency graph that will throw an error that says, "I don't see the variable that you are trying to use." However, declaring a "local" variable that has the same name as a "global" variable does not overwrite the global variable or change the scope the variable. Lines 8-13 is only here to ensure the Apply takes because without any resource TF Cloud will say nothing to do. Line 5 is here so that something actually changes that causes version of the file to change (which triggers a run in TF Cloud).

Here is the end result:

Wait... there's more. What can I get more than the name of workspace? Run a local-exec for all environment variables. Add this resource block to the above code and re-run it.

resource "null_resource" "envs" {
  provisioner "local-exec" {
      command = "printenv"
  }
}

Based on my sampling, it appears that any variables that begins with "TF_VAR_" can be obtained.

TF_VAR_ATLAS_ADDRESS=https://app.terraform.io
TF_VAR_ATLAS_CONFIGURATION_NAME=Lambchop
TF_VAR_ATLAS_CONFIGURATION_SLUG=my_org_name/Lambchop
TF_VAR_ATLAS_CONFIGURATION_VERSION_GITHUB_BRANCH=master
TF_VAR_ATLAS_CONFIGURATION_VERSION_GITHUB_COMMIT_SHA=99999999999
TF_VAR_ATLAS_RUN_ID=run-J2AHxyXbkcT7VJDD
TF_VAR_ATLAS_WORKSPACE_NAME=Lambchop
TF_VAR_ATLAS_WORKSPACE_SLUG=my_org_name/Lambchop
TF_VAR_TFE_RUN_ID=run-J2AHxyXbkcT7VJDD

Here is the TF code to get couple more environment variables

variable "TFE_RUN_ID"{
}
output "TFE_RUN_ID"{
  value = var.TFE_RUN_ID 
}

variable "ATLAS_ADDRESS"{
}
output "ATLAS_ADDRESS"{
  value = var.ATLAS_ADDRESS
}