Lambda read S3 object with S3 trigger

Configuring your Lambda function to read S3 object with S3 upload trigger

Are you new to Lambda and haven't quite got the test case figured out for S3 upload trigger? In the steps below, you'll build a new Python Lambda function from a blueprint and use the CloudWatch log to populate a new test case. This way you don't have to upload just to test your new function.

1. Create a new Lambda Function
2. Select "Use a blueprint"
3. Search for "S3"
4. Select "s3-get-object-python3"
   
5.  Configure it using mostly default settings
1. Function Name: of your choice
2. Create a new role from AWS policy template
3. Role Name: of your choice
4. Select your own trigger bucket
5. Select all object create event
6. Leave prefix and Suffix blank
7. Enable trigger
8. Select Create Function
6.  Uncomment line 12 that begins print('Received event: '
   
7.  Upload a file to the bucket you select in the previous step
8.  From the Lambda function console, go to Monitoring
   
9. Go to View logs in CloudWatch - this will open a new tab into CloudWatch
10. Click the latest and probably ONLY log stream
11. View in TEXT (not rows)
12. Copy the text like below as shown in your log screen into your clipboard
    
13. Now go back to your Lambda function console
14. Up by the top between Action and Test, click on the drop down and select Configure Test Event
    
15. Create new test event (it does not matter what template you choose)
16. Give it a name
17. In the body, paste the string from previous step that you put in your clipboard (overwrite existing text)
    
18. Save and then click Test
19. Your Lambda function will behave as if you've just uploaded that same file again
20. Now go edit the function to do what you want and test easily!

Powershell Update ENI's DNS Tag

Update AWS network interface's Tag with its DNS entry (PowerShell 3.0)

Want to keep a tag that contains the DNS entry of an IP address of your ENIs?

This PowerShell function calls nslookup (Windows native) and adds a tag to ENI with the returned value.

You can also find me here.

################################################
#
# Get all the reserved ENI in the VPC and
#  create a new tag based on the 
#  responding nslookup on the Private IP
#
#
#  (\_/)
#  (>.<)
# (")_(")
#
#################################################
# Get ENIs with ID, Status, PrivateIP, and DNS Tag
$ip_raw = aws ec2 describe-network-interfaces --filter "Name=vpc-id,Values=vpc-xxxxxxx" --query "NetworkInterfaces[*].{ID:NetworkInterfaceId,Status:Status,IP:PrivateIpAddress,DNS:TagSet[?Key=='DNS'].Value}"
# Convert this to PS Object
$ip = $ip_raw | out-string | convertfrom-json
foreach($item in $ip){
# Proceed only if DNS Tag does NOT exist
  if($item.DNS.length -eq 0){
    $error.clear()
    $dns = nslookup $item.ip
    if($error.count -eq 0){
      $name_line = $dns | ?{$_ -like 'Name*'}[0]
      $dns_name = ($name_line.split(":")[1].trim()
    }else{
      $dns_name = "None"
    }
    aws ec2 create-tags --resources $item.ID --tags "Key=DNS,Value=$dns_name"
  }
}

Lambda Update EC2 Tags

Update EC2 Tags using Python (Lambda function)

Simple example demonstrating Python's ability to lookup and update Tags.
You can also find me here. 

import json
import boto3
from datetime import date

# I don't like the dictionary returned by AWS, so I convert it to
# Key:Value pairs
def parse_tags(tag_dict):
    my_dict = {}
    for tag in tag_dict:
        for item in tag:
            if item == 'Key':
                key = tag[item]
            else:
                value = tag[item]
        my_dict[key]= value
    return my_dict

def lambda_handler(event, context):
    today = date.today()
    year = today.strftime("%Y")
    month = today.strftime("%m")
    day = today.strftime("%d")
    #Declare object for all of our ec2 objects in this region
    ec2 = boto3.resource('ec2', region_name='us-east-2')
    #give me all the instances
    instances = ec2.instances.all()
    print('Instances')
    for ins in instances:
        print("Instance Id: ", ins.id)
        ins_tag_dict = {}
        if(ins.tags != None):
            ins_tag_dict = parse_tags(ins.tags)
        #get function of dictionary return None if not found
        name_tag = ins_tag_dict.get('Name')
        if(name_tag == None):
            #Create a name tag for this object
            name_tag = 'Sam'
            ins.create_tags(Tags=[{'Key':'Name','Value': name_tag}])
        #Give me all the volumes for this instance
        volumes = ins.volumes.all()
        for vol in volumes:
            vol_tag_dict = {}
            if(vol.tags != None):
                vol_tag_dict = parse_tags(vol.tags)
            print("Volume Id: ",vol.id)
            # attachment (LIST) has following values:
            ## [{'AttachTime': datetime.datetime(2018, 12, 3, 5, 11, 5, tzinfo=tzlocal()), 'Device': '/dev/xvda', 'InstanceId': 'i-XXXXXXXX', 'State': 'attached', 'VolumeId': 'vol-XXXXXXX', 'DeleteOnTermination': True}]
            # Convert this LIST to DICT
            vol_att_dict = vol.attachments[0]
            vol_device = vol_att_dict.get('Device')
            vol_name_tag = vol_tag_dict.get('Name')
            if(vol_name_tag == None):
                #Create a name tag for this object
                vol_name_tag = name_tag + '_' + vol_device
                vol.create_tags(Tags=[{'Key':'Name','Value': vol_name_tag}])
        #Give ENI names of the EC2 Instance if they are missing name tag
        net_interfaces = ins.network_interfaces
        for eni in net_interfaces:
            print("ENI Id: ",eni.id)
            eni_tag_dict = {}
            if(eni.tag_set != None):
                eni_tag_dict = parse_tags(eni.tag_set)
            eni_name_tag = eni_tag_dict.get('Name')
            if(eni_name_tag == None):
                eni.create_tags(Tags=[{'Key':'Name','Value':name_tag}])
    print('Volumes not in use')
    #give me all the volumes that are not in use
    volumes = ec2.volumes.filter(Filters=[{'Name': 'status', 'Values': ['available']}])
    for vol in volumes:
        print("Volume Id: ",vol.id)
        vol_tag_dict = {}
        if(vol.tags != None):
            vol_tag_dict = parse_tags(vol.tags)
        vol_mode_tag = vol_tag_dict.get('Mode')
        if(vol_mode_tag == None):
            #If Mode tag does not exist, make it Auto mode
            vol.create_tags(Tags=[{'Key':'Mode','Value':'Auto'}])
        vol_expire_tag = vol_tag_dict.get('Expire')
        if(vol_expire_tag == None):
            #If Expire tag does not exist then set it to 7 day from now
            ## we'd have a different function to do the actual cleanup
            new_day = str(int(day) + 7).zfill(2)
            expireDate = year + month + new_day
            vol.create_tags(Tags=[{'Key':'Expire','Value': expireDate}])
    return {
        'statusCode': 200,
        'body': json.dumps('Finished!')
    }

Redirection with JIRA

Redirection with JIRA

Setting up redirect within JIRA to mandate coming from Apache page (on RedHat 7).

1. Install JIRA

Here's the default installation outcome for JIRA 7.13.1:

Installation Directory: /opt/atlassian/jira
Home Directory: /var/atlassian/application-data/jira
HTTP Port: 8080
RMI Port: 8005

2. Install httpd (sudo yum install httpd)

3. Edit the file at:
/var/www/html/index.html
Here's the content:

<html>
<body>
<p>
<a href="http://10.10.0.1:8080/secure/Dashboard.jspa">Go to Jira</a>
</p>
</body>
</html>

4. Start httpd (systemctl start httpd)

5. Edit this file:
/opt/atlassian/jira/atlassian-jira/WEB-INF/urlrewrite.xml

JIRA uses tuckey's urlrewritefilter plugin to Tomcat. So you can refer to tuckey for additional instructions.

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE urlrewrite PUBLIC "-//tuckey.org//DTD UrlRewrite 4.0//EN" "http://www.tuckey.org/res/dtds/urlrewrite4.0.dtd">

<!--
    URL Rewrite files to make issue navigator URL backwards compatible and some other things
    @since JIRA 3.3
-->
<urlrewrite>
    <!-- Caching of static resources -->
    <class-rule class="com.atlassian.jira.plugin.webresource.CachingResourceDownloadRewriteRule"/>
    <!-- @since 5.0 [KickAss]-->
    <rule>
        <from>^/issues(\?.*)?$</from>
        <to type="permanent-redirect">issues/$1</to>
    </rule>
<!--
 Here is our sample rule
-->
    <rule>
       <from>^/secure/Dashboard.jspa$</from>
        <condition name="referer" operator="notequal">^http://10.10.0.1/.*$</condition>
        <condition name="referer" operator="notequal">^http://10.10.0.1:8080/.*$</condition>
        <to type="redirect">http://10.10.0.1</to>
    </rule>
</urlrewrite>

6. You have to restart JIRA for the changes to apply.
Remember, your start/stop script are located under /opt/atlassian/jira/bin

7. To verify that your rules are valid, from localhost browse to http://localhost:8080/rewrite-status. This will output the rewrite rule status.

Setting up User Pool in AWS Cognito

Setting up User Pool in AWS Cognito

It's easy to get started with User Pool in AWS Cognito for your application.

Go to AWS Cognito

Click Manage User Pools


Click Create a user pool (top right corner)


Give it a name and click Review Default


As part of default, only email is required.

Scroll all the way to bottom and click


On the next screen, you should see a notification that the pool was created successfully

Save the Pool Id


You can also retrieve this Pool Id later by selecting the pool and then selecting General settings
 

Click App Clients (sub-category of General settings)


Click add an app client


Give it a name and deselect all options





Create and save App client id


That's it. Now you can use Pool ID and App Client ID in your application, such as config.js.

window._config = {
    cognito: {
        userPoolId: 'us-east-1_xxx', // e.g. us-east-2_uXboG5pAb
        userPoolClientId: '32', // e.g. 25ddkmj4v6hfsfvruhpfi7n4hv
        region: 'us-east-1' // e.g. us-east-2
    },
    api: {
        invokeUrl: 'https://xyz.execute-api.us-east-1.amazonaws.com/prod' // e.g. https://rc7nyt4tql.execute-api.us-west-2.amazonaws.com/prod',
    }
};

Updating S3 Object ACL

Updating AWS S3 Object ACL

Do you have multiple AWS Accounts and did you accidentally upload objects to a bucket of Account A while using Account B's credential? By default without "--acl" flag, the object are still owned by Account B and as Account A, you won't be able to modify them. You could re-upload these files using correct Account credential. Or you can use the below script to modify all object's ACLs in that bucket.

$bucket = "my-bucket"
$bucketname = "s3://" + $bucket 

##These must be 64 digit CANONICAL ID###

$accountA = "XXXXXXXXXXXXXX"
$accountB = "YYYYYYYYYYYYYY"
$output = aws s3 ls $bucketname --recursive

foreach($item in $output){
    $arrayParts = $item.split(" ")
    #Get the last part of the object, which is the file name
    $object = $arrayParts[$arrayParts.count - 1]
    #If the file name then it does not end in "/", otherwise it is a prefix
    if($object[$object.length - 1] -ne "/"){
        $output2 = $bucketname + "/" + $object
        ##OPTION A###
        ##Give full control to bucket owner
        aws s3api put-object-acl --bucket $bucket --key $object --acl bucket-owner-full-control

        ##OPTION B###
        ##Give full control to both accounts
        #aws s3api put-object-acl --bucket $bucket --key $object --grant-full-control "id=$accountA,id=$accountB"
    }
}