Saturday, March 9, 2019

Lambda Update EC2 Tags

Update EC2 Tags using Python (Lambda function)


Simple example demonstrating Python's ability to lookup and update Tags.
You can also find me here


import json
import boto3
from datetime import date

# I don't like the dictionary returned by AWS, so I convert it to
# Key:Value pairs
def parse_tags(tag_dict):
    my_dict = {}
    for tag in tag_dict:
        for item in tag:
            if item == 'Key':
                key = tag[item]
            else:
                value = tag[item]
        my_dict[key]= value
    return my_dict

def lambda_handler(event, context):
    today = date.today()
    year = today.strftime("%Y")
    month = today.strftime("%m")
    day = today.strftime("%d")
    #Declare object for all of our ec2 objects in this region
    ec2 = boto3.resource('ec2', region_name='us-east-2')
    #give me all the instances
    instances = ec2.instances.all()
    print('Instances')
    for ins in instances:
        print("Instance Id: ", ins.id)
        ins_tag_dict = {}
        if(ins.tags != None):
            ins_tag_dict = parse_tags(ins.tags)
        #get function of dictionary return None if not found
        name_tag = ins_tag_dict.get('Name')
        if(name_tag == None):
            #Create a name tag for this object
            name_tag = 'Sam'
            ins.create_tags(Tags=[{'Key':'Name','Value': name_tag}])
        #Give me all the volumes for this instance
        volumes = ins.volumes.all()
        for vol in volumes:
            vol_tag_dict = {}
            if(vol.tags != None):
                vol_tag_dict = parse_tags(vol.tags)
            print("Volume Id: ",vol.id)
            # attachment (LIST) has following values:
            ## [{'AttachTime': datetime.datetime(2018, 12, 3, 5, 11, 5, tzinfo=tzlocal()), 'Device': '/dev/xvda', 'InstanceId': 'i-XXXXXXXX', 'State': 'attached', 'VolumeId': 'vol-XXXXXXX', 'DeleteOnTermination': True}]
            # Convert this LIST to DICT
            vol_att_dict = vol.attachments[0]
            vol_device = vol_att_dict.get('Device')
            vol_name_tag = vol_tag_dict.get('Name')
            if(vol_name_tag == None):
                #Create a name tag for this object
                vol_name_tag = name_tag + '_' + vol_device
                vol.create_tags(Tags=[{'Key':'Name','Value': vol_name_tag}])
        #Give ENI names of the EC2 Instance if they are missing name tag
        net_interfaces = ins.network_interfaces
        for eni in net_interfaces:
            print("ENI Id: ",eni.id)
            eni_tag_dict = {}
            if(eni.tag_set != None):
                eni_tag_dict = parse_tags(eni.tag_set)
            eni_name_tag = eni_tag_dict.get('Name')
            if(eni_name_tag == None):
                eni.create_tags(Tags=[{'Key':'Name','Value':name_tag}])
    print('Volumes not in use')
    #give me all the volumes that are not in use
    volumes = ec2.volumes.filter(Filters=[{'Name': 'status', 'Values': ['available']}])
    for vol in volumes:
        print("Volume Id: ",vol.id)
        vol_tag_dict = {}
        if(vol.tags != None):
            vol_tag_dict = parse_tags(vol.tags)
        vol_mode_tag = vol_tag_dict.get('Mode')
        if(vol_mode_tag == None):
            #If Mode tag does not exist, make it Auto mode
            vol.create_tags(Tags=[{'Key':'Mode','Value':'Auto'}])
        vol_expire_tag = vol_tag_dict.get('Expire')
        if(vol_expire_tag == None):
            #If Expire tag does not exist then set it to 7 day from now
            ## we'd have a different function to do the actual cleanup
            new_day = str(int(day) + 7).zfill(2)
            expireDate = year + month + new_day
            vol.create_tags(Tags=[{'Key':'Expire','Value': expireDate}])
    return {
        'statusCode': 200,
        'body': json.dumps('Finished!')
    }

Monday, February 4, 2019

Redirection with JIRA

Redirection with JIRA

Setting up redirect within JIRA to mandate coming from Apache page (on RedHat 7).

1. Install JIRA

Here's the default installation outcome for JIRA 7.13.1:

Installation Directory: /opt/atlassian/jira
Home Directory: /var/atlassian/application-data/jira
HTTP Port: 8080
RMI Port: 8005

2. Install httpd (sudo yum install httpd)

3. Edit the file at:
/var/www/html/index.html
Here's the content:

<html>
<body>
<p>
<a href="http://10.10.0.1:8080/secure/Dashboard.jspa">Go to Jira</a>
</p>
</body>
</html>

4. Start httpd (systemctl start httpd)

5. Edit this file:
/opt/atlassian/jira/atlassian-jira/WEB-INF/urlrewrite.xml

JIRA uses tuckey's urlrewritefilter plugin to Tomcat. So you can refer to tuckey for additional instructions.


<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE urlrewrite PUBLIC "-//tuckey.org//DTD UrlRewrite 4.0//EN" "http://www.tuckey.org/res/dtds/urlrewrite4.0.dtd">

<!--
    URL Rewrite files to make issue navigator URL backwards compatible and some other things
    @since JIRA 3.3
-->
<urlrewrite>
    <!-- Caching of static resources -->
    <class-rule class="com.atlassian.jira.plugin.webresource.CachingResourceDownloadRewriteRule"/>
    <!-- @since 5.0 [KickAss]-->
    <rule>
        <from>^/issues(\?.*)?$</from>
        <to type="permanent-redirect">issues/$1</to>
    </rule>
<!--
 Here is our sample rule
-->
    <rule>
       <from>^/secure/Dashboard.jspa$</from>
        <condition name="referer" operator="notequal">^http://10.10.0.1/.*$</condition>
        <condition name="referer" operator="notequal">^http://10.10.0.1:8080/.*$</condition>
        <to type="redirect">http://10.10.0.1</to>
    </rule>
</urlrewrite>

6. You have to restart JIRA for the changes to apply.
Remember, your start/stop script are located under /opt/atlassian/jira/bin

7. To verify that your rules are valid, from localhost browse to http://localhost:8080/rewrite-status. This will output the rewrite rule status.

Wednesday, January 9, 2019

Setting up User Pool in AWS Cognito

Setting up User Pool in AWS Cognito

It's easy to get started with User Pool in AWS Cognito for your application.


Go to AWS Cognito

Click Manage User Pools


Click Create a user pool (top right corner)


Give it a name and click Review Default


As part of default, only email is required.

Scroll all the way to bottom and click


On the next screen, you should see a notification that the pool was created successfully

Save the Pool Id


You can also retrieve this Pool Id later by selecting the pool and then selecting General settings
 

Click App Clients (sub-category of General settings)


Click add an app client


Give it a name and deselect all options





Create and save App client id


That's it. Now you can use Pool ID and App Client ID in your application, such as config.js.


window._config = {
    cognito: {
        userPoolId: 'us-east-1_xxx', // e.g. us-east-2_uXboG5pAb
        userPoolClientId: '32', // e.g. 25ddkmj4v6hfsfvruhpfi7n4hv
        region: 'us-east-1' // e.g. us-east-2
    },
    api: {
        invokeUrl: 'https://xyz.execute-api.us-east-1.amazonaws.com/prod' // e.g. https://rc7nyt4tql.execute-api.us-west-2.amazonaws.com/prod',
    }
};
















Wednesday, January 2, 2019

Updating S3 Object ACL

Updating AWS S3 Object ACL

Do you have multiple AWS Accounts and did you accidentally upload objects to a bucket of Account A while using Account B's credential? By default without "--acl" flag, the object are still owned by Account B and as Account A, you won't be able to modify them. You could re-upload these files using correct Account credential. Or you can use the below script to modify all object's ACLs in that bucket.



$bucket = "my-bucket"
$bucketname = "s3://" + $bucket 
##These must be 64 digit CANONICAL ID###
$accountA = "XXXXXXXXXXXXXX"
$accountB = "YYYYYYYYYYYYYY"
$output = aws s3 ls $bucketname --recursive

foreach($item in $output){
    $arrayParts = $item.split(" ")
    #Get the last part of the object, which is the file name
    $object = $arrayParts[$arrayParts.count - 1]
    #If the file name then it does not end in "/", otherwise it is a prefix
    if($object[$object.length - 1] -ne "/"){
        $output2 = $bucketname + "/" + $object
        ##OPTION A###
        ##Give full control to bucket owner
        aws s3api put-object-acl --bucket $bucket --key $object --acl bucket-owner-full-control

        ##OPTION B###
        ##Give full control to both accounts
        #aws s3api put-object-acl --bucket $bucket --key $object --grant-full-control "id=$accountA,id=$accountB"
    }
}

Monday, December 31, 2018

PowerShell Move Eventlogs to S3

Moving Eventlogs to S3

This can also be done to any remote location. And in the code below, I export as CSV, but you can also move as CAB files, but I prefer to be able to natively read these files without extracting them first. On most servers, this is a scheduled task that is set to run every hour because how fast our Security logs fill up.

Here is a sample of "get-eventlog -list" output


  Max(K) Retain OverflowAction        Entries Log                                                                                                                        
  ------ ------ --------------        ------- ---                                                                                                                        
  20,480      0 OverwriteAsNeeded          25 Application                                                                                                                
  20,480      0 OverwriteAsNeeded           0 HardwareEvents                                                                                                             
     512      7 OverwriteOlder              0 Internet Explorer                                                                                                          
  20,480      0 OverwriteAsNeeded           0 Key Management Service                                                                                                     
  20,480      0 OverwriteAsNeeded     225,262 Security                                                                                                                   
  20,480      0 OverwriteAsNeeded          45 System                                                                                                                     
  15,360      0 OverwriteAsNeeded         450 Windows PowerShell 

Actual Code to backup Eventlogs


#####################
#
# Export all eventlogs as CSV
# Clears logs after export
#
#####################
#This command will gather all the available Logs (not the actual logs themselves)
$evLogs = get-eventlog -list
$currentTime = $(get-date)
$thisDate = $currentTime.GetDateTimeFormats()[5]
#get date/time information and pad the numbers
$year = ($currentTime.Year | out-string).trim().padleft(4,"0")
$month = ($currentTime.Month | out-string).trim().padleft(2,"0")
$day = ($currentTime.Day | out-string).trim().padleft(2,"0")
$hour = ($currentTime.hour | out-string).trim().padleft(2,"0")
$min = ($currentTime.Minute | out-string).trim().padleft(2,"0")

try{
    if((test-path "e:") -eq $false){
        $rootDir = "c:\temp"
    }else{
        $rootDir = "e:\temp"
    }

    if((test-path $rootDir) -eq $false){
        mkdir $rootDir -Force
    }
    ##This is my target bucket
    $targetBucket = "s3://my-server-logs" 
    $targetPrefix = $targetBucket + "/" + $env:COMPUTERNAME + "/" + $year + "/" + $month + "/" + $day

    foreach($log in $evLogs){
        if($log.entries.count -gt 0){
            $filename = $thisdate + "-" + $hour + $min + "-" + $log.log + ".csv"
            $sourcefilename = $rootDir + "\" + $filename
            $targetfilename = $targetPrefix + "/" + $filename
            $events = get-eventlog -log $log.Log
            $events | Export-Clixml $sourcefilename
            if(Test-Path $sourcefilename){
                Clear-EventLog -logname $log.Log    
                aws s3 mv $sourcefilename $targetfilename
            }
        }
    }
}catch{
    ##Put error catching here...
}

PowerShell Cleanup C Drive

Cleaning C Drive

I do this on a Windows 2012R2 server.

Move the archived eventlogs to S3 bucket

#Archives are located here by default
$archiveFiles = Get-ChildItem -path C:\windows\system32\winevt -include "Archive*" -Recurse
#This is the S3 bucket where we'll keep these logs
$targetBucket = "s3://my-server-logs"
#We are going to organize these logs files under the computer name and date stamp
$targetPrefix = $targetBucket + "/" + $env:COMPUTERNAME
foreach($item in $archiveFiles){
    $splitoutput = $item.name.split("-")
    $year = $splitoutput[2]
    $month = $splitoutput[3]
    $day = $splitoutput[4]
    if(($year -match "\d{4}") -and ($month -match "\d{2}") -and ($day -match "\d{2}")){
        $targetFile = $targetPrefix + "/" + $year + "/" + $month + "/" + $day + "/" + $item.name
        aws s3 mv $item.fullname $targetFile
    }
}


Cleanup applied service packs and updates

This will prevent rollback ability so only do this if you've verified patches didn't break anything.

##Remove superseded and unused system files
dism.exe /online /Cleanup-Images /StartComponentCleanup
##All existing service packs and updates cannot be uninstalled after this update
dism.exe /online /Cleanup-Images /StartComponentCleanup /ReserBase
##Service packs cannot be uninstalled after this command
dism.exe /online /Cleanup-Images /SPSuperseded


Relocate Software Distribution Directory



##Stop Windows Update Service
net stop wuauserv
##Rename current software distribution directory
rename-item C:\windows\SoftwareDistribution SoftwareDistribution.old
##Create a new location for this distribution directory
mkdir E:\Windows-SoftwareDistribution
##Make a link 
cmd /c mklink /J C:\Windows\SoftwareDistribution "E:\Windows-SoftwareDistribution"
##Start service
net start wuauserv
rmdir C:\windows\SoftwareDistribution.old -confirm

AWS WAF log4j query

How to query AWS WAF log for log4j attacks 1. Setup your Athena table using this instruction https://docs.aws.amazon.com/athena/latest/ug/wa...