Using Boto3 Paginator

Using Boto3 Paginator

If you are retrieving any data from AWS using Boto3, you should make a habit of using Paginator for the day when your data-set gets too large for a single call to handle.

Here's a simple illustration on how to lookup paginator arguments and returns and how to use it in your code.

Go to Boto3 Docs
Every service has a paginators. Write your code based on the class definition as shown above.
Here's the code:

thisClient = boto3.client("iam", aws_access_key_id=credentials['AccessKeyId'],
                                 aws_secret_access_key=credentials['SecretAccessKey'],
                                 aws_session_token=credentials['SessionToken'])
paginator = thisClient.get_paginator('list_users')
    response_iterator = paginator.paginate()
    for response in response_iterator:
        for key in response.get('Users'):
            thisUserId = key.get('UserId')
            thisUserName = key.get('UserName')

Managing Terraform State file

Terraform Notes

Moving resources and backing up tf state

This will illustrate process of migrating resources from one state to another as well as backing up and reverting to different states. 

In your working directory, you should have at least terraform.tfstate. If you have defined an external backend, then this file may be cached under .terraform directory (and your actual changes are happening on remote location). 

if your main.tf is this:

variable "timestamps" {
  type    = bool
  default = false
}

resource "null_resource" "exampleA" {
  triggers = {
    key = formatdate("YYYY-MM-DD hh:mm:ssZZZZZ", timestamp())
  }
}

Then your intial terraform.tfstate would look like this:

{
  "version": 4,
  "terraform_version": "0.12.24",
  "serial": 1,
  "lineage": "d0427b0f-6517-25bb-f708-6666666666",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "null_resource",
      "name": "exampleA",
      "provider": "provider.null",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "8139684392621207633",
            "triggers": {
              "key": "2020-04-23 17:41:12+00:00"
            }
          },
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

To get list of all your resource, you can run "terraform state list"

From the output from above, you can use the mv command to move resources to be managed by different terraform state. You can also use this command to refactor the name of the resource.

terraform state mv 

-backup='terraform.backup.tfstate' 

-state-out='terraform.new.tfstate' 

 null_resource.exampleA 

 null_resource.exampl

The above command will create two additional terraform state files.

terraform.tfstate

{
  "version": 4,
  "terraform_version": "0.12.24",
  "serial": 2,
  "lineage": "d0427b0f-6517-25bb-f708-6666666666",
  "outputs": {},
  "resources": []
}

terraform.new.tfstate

{
  "version": 4,
  "terraform_version": "0.12.24",
  "serial": 1,
  "lineage": "c7e1be5e-5363-39be-7078-6666666666",  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "null_resource",
      "name": "exampleB",
      "provider": "provider.null",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "8139684392621207633",
            "triggers": {
              "key": "2020-04-23 17:41:12+00:00"
            }
          },
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

terraform.backup.tfstate

{
  "version": 4,
  "terraform_version": "0.12.24",
  "serial": 1,
  "lineage": "d0427b0f-6517-25bb-f708-6666666666",  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "null_resource",
      "name": "exampleA",
      "provider": "provider.null",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "8139684392621207633",
            "triggers": {
              "key": "2020-04-23 17:41:12+00:00"
            }
          },
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

Note that lineage of terraform.tfstate and terraform.backup.tfstate is going to be identical.

If you run terraform state list now you'll get nothing.

To go back to previous tf state, you can run the command in reverse where source (-state) is the new terraform state file and destination (-state-out) is the original state file.

terraform state mv 
-backup='terraform.backup2.tfstate' 
-state-out='terraform.tfstate' 
-state='terraform.new.tfstate'
 null_resource.exampleB 
 null_resource.exampleA

If this is just a local state, you can also just rename terraform.backup.tfstate to terraform.tfstate.

Now if you run terraform state list again you'll get your resource back.

Terraform for_each caution

Terraform Notes

For_Each Loop Caution

When using for_each, be sure to be mindful of what you want these new resources to be indexed as. Because if you ever try to change the index name, the resources must be destroyed and recreated. 

Example map variable

variable "account_map"{
    default = {
        "TEST-A-Key" = {
            "name" = "TEST-A-Name"
            "email" = "TEST-A@me.com"
        },
        "TEST-B-Key" = {
            "name" = "TEST-B-Name"
            "email" = "TEST-B@me.com"
        }
    }
}

When you create a resource based on the above map, you get this:

resourceType.name["TEST-A-Key"]
resourceType.name["TEST-B-Key"]

Unfortunately, in the above example (and in my real world case), I've tied the index name to be same as the name of the account. So when I went to updated the account name (the above variable is created from a file), it attempted to delete all my account resources. So now my only choice is to delete state of these resources and re-import OR live with mis-matched account name and index string of the resources.

So take caution and use some index string that describes the variable, not the specific content. Or maybe you don't mind destroy and create whenever you want to change a name. 

AWS Custom Config Rule from Org

Deploy AWS Config Rule from Org

If you have AWS Org configured, you can deploy Config Rule from a single location out to all the accounts in the same Org. You can do this for both AWS Managed and your Custom Rules. 

References:

AWS Dev Guide: config-rule-multi-account-deployment

Details:

• Max 150 rules
• All necessary permissions and roles must be pre-configured

Advantages:

• Deploy from central location
• Does not allow accounts from editing the rules

Setup

Setup Root Account

• Configure Aggregator
• Configure S3 Logging Bucket for Config* 
• Configure SNS Topic*
• Configure Lambda Role
• Your usual Lambda Role permissions
• Ability to Assume Role to the Recorder Role name in all accounts (this can be extracted from "executionRoleArn" of the Event object)
• Configure Lambda Function
• Need to allow this function to be executed by all accounts in this Org

Setup All Accounts

• Configure Recorder Role 
• use same name across all accounts
• need necessary permission to 
• write to Config 
• write to S3 Logging Bucket in Root Account*
• write to SNS Topic in Root Account*
• Trust Lambda Role in Root Account to AssumeRole
• and read/edit necessary resources for it to do what you want
• Turn on Recording

*The S3 and SNS does not have to be in Root Account, just put it somewhere common

Execution

AWS Doc API PutOrganizationConfigRule