Technical references: 2017

Wednesday, September 6, 2017

AWS Java SDK Example

Connecting to AWS and printing all volumes

public class java-ec2-example{
    public static void main(String[] args){
        String myRegion = "us-east-1";
        String accessKey = "xxxx";
        String secretKey = "xxxx";
        String sessionTk = "xxxx";
        
        //For programmatic access user
        BasicAWSCredentials awsCreds =  new BasicAWSCredentials (accessKey, secretKey);
        
        //For temporary credential
        BasicSessionCredentials sesCreds = new BasicSessionCredentials (accessKey, secretKey, sessionTk);

        //For other credential (e.g. Instance IAM, Environment Variable, etc) leave this as null 
        AWSStaticCredentialsProvider myCredential = null;

        //For this example, we use session credential
        myCredential = new AWSStaticCredentialsProvider(sesCreds);

        AmazonEC2 ec2 = AmazonEC2ClientBuilder.standard()
                     .withCredentials(myCredential)
                     .withRegion(myRegion)
                     .build();

        List<Volume> volumes = ec2.describeVolumes(new DescribeVolumesRequest()).getVolumes();
        for(int i=0;i<volumes.size();i++){
            System.out.println(volumes.get(i));
        }
    }
}

Lines 9-18 are various means to connect to AWS. If you want to rely on the environment variables, you can just comment out line 18 and leave myCredentials as null.

Line 20 is used to build client connector to EC2. Similiar builders are available for all AWS services.

Line 25 returns all the volumes in your account.

Line 26 loops and prints all the volumes.

Wednesday, August 30, 2017

AWS CLI Sample - Find latest snapshot

AWS CLI Sample

Find the most recent snapshots of a volume and update a tag on it

$volumeid = "vol-0000000000000000000"
$output = aws ec2 describe-snapshots `
   --owner-id 4444444444444 `
   --filters "Name=volume-id,Values='$volumeid'" `
   --query 'Snapshots[].{ID:SnapShotId,StartTime:StartTime}'
$output2 = $output | out-string | convertfrom-json
$highDate = "0"
$highID = "0"
foreach($item in $output2){
   if($item.StartTime -gt $highDate){
      $highDate = $item.StartTime
      $highID = $item.ID
   }
}
write-host $highID
$tag_value = "New Stamp (" + $(get-date).GetDateTimeFormats()[93] + ")"
aws ec2 create-tags --resources $highID --tags "Key='Bookmark',Value='$tag_value'"

Lines 2-5 is your standard describe snapshot command. Be sure to include the account number (owner-id) when we're filtering. The query is to limit the data returned since we don't need to know everything about each snapshots.

Lines 9-14 is looping through all the snapshots returned to get the newest one.

Line 16 is just to put a date stamp. GetDateTimeFormats()[93] returns "yyyy-MM-dd hh:mm:ss" date format.

Sunday, August 27, 2017

AWS CloudFormation Sample

AWS CloudFormation - Using Parameters

Parameters are input arguments to CloudFormation script. Items like InterfaceID and InstanceProfile are custom parameters where we define the possible values. NameTag is a customer parameter where input values are open-ended with character limit/type limits. The remainder are AWS Supported Parameter Types, more can be found here.

"Parameters":{
        "ImageId":{
            "Description":"Image ID",
            "Type":"String",
            "Default":"ami-284f1a3e",
            "AllowedValues":[
              "ami-284f1a3e",
              "ami-36174e4d"
            ],
            "ConstraintDescription":""

        },
        "InstanceType":{
            "Description":"Instance Type",
            "Type":"String",
            "Default":"t2.small",
            "AllowedValues":[
       "t2.nano",
       "t2.micro",
       "t2.small",
       "t2.medium",
       "t2.large",
       "t2.xlarge",
       "t2.2xlarge"
            ],
            "ConstraintDescription":""
        },
        "KeyName":{
            "Description":"Key Name",
            "Type":"AWS::EC2::KeyPair::KeyName",
            "ConstraintDescription":""
        },
        "InterfaceID":{
            "Description":"Network Interface ID",
            "Type":"String",
            "Default":"eni-db0e3c7a",
            "AllowedValues":[
              "eni-9158e145",
              "eni-db0e3c7a"
            ],
            "ConstraintDescription":""
        },
        "NameTag":{
            "Description":"Name Tag",
            "Type":"String",
            "Default":"NameTag",
            "MinLength":"1",
            "MaxLength":"10",
            "AllowedPattern":"[a-zA-Z][a-zA-Z0-9]*",
            "ConstraintDescription":""
        },
        "InstanceProfile":{
            "Description":"Instance Profile",
            "Type":"String",
            "Default":"read_only_instance",
            "AllowedValues":[
              "read_only_instance",
              "admin_instance"
            ],
            "ConstraintDescription":""
        }
    }

In the Resources, we reference the above parameters using keyword, "Ref"

"Resources": {
        "MyInstance": {
            "Type": "AWS::EC2::Instance",
            "Properties": {
                "ImageId": {
                    "Ref": "ImageId"
                },
                "InstanceType": {
                    "Ref": "InstanceType"
                },
                "KeyName": {
                    "Ref": "KeyName"
                },
                "NetworkInterfaces": [
                    {
                        "NetworkInterfaceId": 
                            {
                            "Ref": "InterfaceID"
                            },
                        "DeviceIndex":"0"
                    }
                ],
                "Tags": [
                    {
                        "Key": "Name",
                        "Value": {
                            "Ref": "NameTag"
                        }
                    }
                ]
            },
            "Metadata": {
                "AWS::CloudFormation::Designer": {
                    "id": "4b94664e-1e18-400c-9733-ff095ff6e854"
                }
            }
        }
    }

The whole thing looks like this:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters":{
        "ImageId":{
            "Description":"Image ID",
            "Type":"String",
            "Default":"ami-284f1a3e",
            "AllowedValues":[
              "ami-284f1a3e",
              "ami-36174e4d"
            ],
            "ConstraintDescription":""

        },
        "InstanceType":{
            "Description":"Instance Type",
            "Type":"String",
            "Default":"t2.small",
            "AllowedValues":[
       "t2.nano",
       "t2.micro",
       "t2.small",
       "t2.medium",
       "t2.large",
       "t2.xlarge",
       "t2.2xlarge"
            ],
            "ConstraintDescription":""
        },
        "KeyName":{
            "Description":"Key Name",
            "Type":"AWS::EC2::KeyPair::KeyName",
            "ConstraintDescription":""
        },
        "InterfaceID":{
            "Description":"Network Interface ID",
            "Type":"String",
            "Default":"eni-db0e3c7a",
            "AllowedValues":[
              "eni-9158e145",
              "eni-db0e3c7a"
            ],
            "ConstraintDescription":""
        },
        "NameTag":{
            "Description":"Name Tag",
            "Type":"String",
            "Default":"NameTag",
            "MinLength":"1",
            "MaxLength":"10",
            "AllowedPattern":"[a-zA-Z][a-zA-Z0-9]*",
            "ConstraintDescription":""
        },
        "InstanceProfile":{
            "Description":"Instance Profile",
            "Type":"String",
            "Default":"read_only_instance",
            "AllowedValues":[
              "read_only_instance",
              "admin_instance"
            ],
            "ConstraintDescription":""
        }
    },
    "Metadata": {
        "AWS::CloudFormation::Designer": {
            "4b94664e-1e18-400c-9733-ff095ff6e854": {
                "size": {
                    "width": 60,
                    "height": 60
                },
                "position": {
                    "x": 304,
                    "y": 225
                },
                "z": 0
            }
        }
    },
    "Resources": {
        "MyInstance": {
            "Type": "AWS::EC2::Instance",
            "Properties": {
                "ImageId": {
                    "Ref": "ImageId"
                },
                "InstanceType": {
                    "Ref": "InstanceType"
                },
                "KeyName": {
                    "Ref": "KeyName"
                },
                "NetworkInterfaces": [
                    {
                        "NetworkInterfaceId": 
                            {
                            "Ref": "InterfaceID"
                            },
                        "DeviceIndex":"0"
                    }
                ],
                "Tags": [
                    {
                        "Key": "Name",
                        "Value": {
                            "Ref": "NameTag"
                        }
                    }
                ]
            },
            "Metadata": {
                "AWS::CloudFormation::Designer": {
                    "id": "4b94664e-1e18-400c-9733-ff095ff6e854"
                }
            }
        }
    }
}

When you create stack from this:

Thursday, August 10, 2017

Powershell Network Tools

Few Powershell Network tools

This is when you are in an environment that won't let you install software. I've collected these from various sources. I use them often enough to keep it handy here.

Check for TCP Listener:

$socket = new-object net.sockets.tcpclient
$socket.connect("www.google.com",443)
$socket.connected
$socket = $null

Check for TCP Connections on your machine:

This is more than you really need, but it'll print a table of established connections every 5 seconds. If you just want the raw data and all the available connection state, just run lines 2 and 3.

do{
 $tcpproperties=[System.Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties()
 $connections=$tcpproperties.GetActiveTcpConnections()
 $output=@()
 foreach($conn in $connections){
   if($conn.State -eq 'Established'){
     $outputObj = New-Object -typename PSObject
     $outputObj | Add-Member -MemberType NoteProperty -Name "LocalAddress" -Value $conn.LocalEndPoint.Address 
     $outputObj | Add-Member -MemberType NoteProperty -Name "LocalPort" -Value $conn.LocalEndPoint.Port
     $outputObj | Add-Member -MemberType NoteProperty -Name "RemoteAddress" -Value $conn.RemoteEndPoint.Address
     $outputObj | Add-Member -MemberType NoteProperty -Name "RemotePort" -Value $conn.RemoteEndPoint.Port
     $outputObj | Add-Member -MemberType NoteProperty -Name "State" -Value $conn.State
     $output=$output + $outputObj
   }
 }
 $output | format-table
 sleep 5
}while($true)

Check for UDP Listener

For this one, I'm also illustrating a listener too. Good for troubleshooting firewall.

Receiver

Run this portion on your receive end and wait for traffic.

$udpObject = new-object system.net.sockets.udpclient(1433)
$udpObject.client.ReceiveTimeout = 10000
do{
  $remoteEndPoint = new-object System.Net.IPEndPoint([System.Net.IPAddress]::Any,0)
  $receiveBytes = $udpObject.Receive([ref]$remoteEndPoint)
  $a = New-Object System.Text.AsciiEncoding
  [String]$returnData = $a.GetString($receiveBytes)
  Write-host "Received: $($returnData.ToString())"
  Write-host "Sent from: $($remoteEndPoint.Address.ToString()) on their port: $($remoteEndPoint.port.ToString())"
  $ans = read-host "More (y/n)?"
}while($ans -ne 'n')
$udpObject.close()

Sender

Then run this from your remote machine to see the information show up on the remote end.

$udpObject = new-object system.net.sockets.udpclient(1433)
$udpObject.connect("10.10.10.1",1433)
$a = new-object System.Text.AsciiEncoding
$byte = $a.GetBytes("$(get-date)")
[void]$udpObject.send($byte,$byte.length)
$udpObject.close()

Check for UDP Connection

1 2	$udpProperties = [System.Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties() $udpProperties.GetActiveUDPListners()

Remote PowerShell Session

Again, more than you need, but incase you don't want to have to type in your password every time.

$servername = "mydesktop1"
$username = "domain\username"
$password = "password"
$secPassword = convertTo-SecureString $password -AsPlainText -Force
$mycred = New-Object System.Management.Automation.PSCredential($username,$secPassword)
enter-pssession -Credential $mycred -Computername $servername

Send Email

Good way to test your SMTP Server.

$SMTP = "mail.mycompany.net"
$Subject = "Test from $env:Computername"
$toaddress = "myemail@mycompany.net"
$fromaddress = $env:Computername + "@mycompany.net"
$body = "this message was sent via $SMTP at $(get-date)."
send-mailmessage -smtpserver $SMTP -to $toaddress -from $fromaddress -subject $subject -body $body

Wednesday, August 9, 2017

CFML Javascript Update One to Many relationship tables

CFML/Javascript Update Table Page

You have list of users. Those users can belong to 0 to many roles. I needed a simple way to update this table. This is my first approach.

Here's the basic view of the tables and their relationship.

Here's the header portion of the code with the two Javascript functions and the call to my first database table.
alterRole populates a text box that will be submitted when we get ready to save the changes.
resetButton re-enabled all the pressed Add/Delete buttons so that the user can begin again.

<html>
<title>Edit Roles</title>
<head>
<script type ="text/javascript">
function alterRole(action,roleId){
  document.getElementById(roleId).disabled="disabled";
  if(action == 1){
    var txt = document.getElementById("Add").value;
    txt = txt + ";" + roleId;
    document.getElementById("Add").value = txt;
  }else{
    var txt = document.getElementById("Del").value;
    txt = txt + ";" + roleId;
    document.getElementById("Del").value = txt;
  }
}

function resetButton(){
  var inputs = documents.getElementByTagName("BUTTON");
  for (var i=0; i<inputs.length;i++){
    if (inputs[i].type == 'button'){
      input[i].disabled = false;
    }
  }
  document.getElementById("Add").value = "";
  document.getElementById("Del").value = "";
}
</script>

<cfquery name="users" datasource="MYDB">
  select PID, my_UserID from user_list order by userid
</cfquery>
<cfset user_id = 1>
</head>
<cfheader name = "Expires" value "#Now()#">

Below is the first part of the body. This portion is executed when we press Load (user roles) or Save. If we pressed Load then the page loads the currently selected Roles (and available) for the User we selected. If we press Save, then the page inserts into or remove from the intermediate table the UserIDs and RoleIDs. The Form.Add and Form.Del is populated in the last portion of the code.

<body><form action="edit_roles.cfm" method="post">
<cfif isDefined("form.load") or isDefined("form.save")>
  <cfset thisUserID = #form.user_id#>
  <input name="UserName" type="text" value='<cfoutput>#form.user_id#</cfoutput>' size="10" hidden>
  <cfif isDefined("form.save")>
    <cfif len(#form.Add#) gt 0>
      <cfset this1 = #form.Add#>
      <cfset this1 = right(this1,len(this1)-1)>
      <cfset this2 = this1.split(";")>
      <cfloop array = "#this2#" index = "item">
        <cfquery name = "user_to_role" datasource="MYDB">
          insert into user_to_role(user_pid,roleid) values (#thisUserId#,'#item#')
        </cfquery>
      </cfloop>
    </cfif>
    <cfif len(#form.Del#) gt 0>
      <cfset this1 = #form.Del#>
      <cfset this1 = right(this1,len(this1)-1)>
      <cfset this2 = this1.split(";")>
      <cfloop array = "#this2#" index = "item">
        <cfquery name ="user_to_role" datasource="MYDB">
          delete from user_to_role where user_pid= #thisUserId# and roleid = '#item#'
        </cfquery>
      </cfloop>
    </cfif>    
<cfelse>
  <cfset thisUserId = 0>
  <input name ="UserName" type ="text" value ="" size ="10" hidden>
</cfif>

This section populates the entire list of Users and pre-selects the user if we're coming from Save action. This section also has the query that contains the entire list of Roles available and Roles that are already defined for the user.

<h2>User Roles Management</h2>
<h3>Users:
<select name ="user_id" id ="user_id">
  <cfoutput query="users">
    <cfif #PID# eq #thisuserid#>
      <option value="#PID#" selected>#my_USERID#</option>
    <cfelse>
      <option value="#PID#">#my_USERID#</option>
    </cfif>
  </cfoutput>
</selected>
<input name ="Load" type ="Submit" value ="Load"/>
<input name ="Save" type ="Submit" value ="Save"/>
</h3><br>
<cfif #thisuserid# gt 0>
  <cfquery name="user_to_roles" datasource ="MYDB">
    select user_pid, rolename, roles.roleId, ownerid 
    from roles left outer join user_to_role 
    on roles.roleid = user_to_role.roleid 
    and user_pid = #thisuserid# where roletype = 'User' order by rolename
</cfquery>

This section populates a table with the available Roles. If the selected User is already associated with a role then we provide the option to remove, otherwise we have the option add. When we add or delete a role from the selected User, then the RoleID is added to the appropriate text box that will be submitted when we press Save.

<table id ="roles_table">
  <thead>
    <tr>
      <th>RoleName</th>
      <th>Owner</th>
      <th><button type="button" onClick="javascript:resetbutton();">Reset Buttons</button></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <cfoutput query = "user_to_roles">
    <tr>
      <th>#RoleName#</th>
      <th>#Ownerid#</th>
      <cfif len(#user_pid#)>
        <button type="button" onClick="javascript:alterRole(0,'#roleid#');" value ="#roleid#" id = "#roleid#" name = "del_btn">
        Remove</button>
      <cfelse>
        <button type="button" onClick="javascript:alterRole(1,'#roleid#');" value ="#roleid#" id = "#roleid#" name = "add_btn">
        Add</button>
      </cfif>
      <th></th>
      <th></th>
    </tr>
    </cfoutput>
  </tbody>
  </table>
</cfif>
<input name ="Add" id ="Add" type ="text" value ="" size ="100" hidden>
<input name ="Del" id ="Del" type ="text" value ="" size ="100" hidden>
</form>
</body>
</html>

For next version, I plan on using AJAX on every Add/Delete action so that we don't have to submit the entire page. Also, some IF branches can be slimmed down if we move some common actions into a function in CFC.

---

Thursday, August 3, 2017

Perl Webpage status checker

How to check webpage status using Perl

We have several web pages that we needs to be monitored from outside of our work network especially after a maintenance window. I didn't want to do this manually (ever again), so I wrote a Perl script to run on my Linux server (laptop) from home.

This is the header portion. I imported these modules to be used by my script. Line 3 turns any expression that is deemed difficult to debug into error. And line 4 enabled optional warnings.

#!/usr/local/bin/perl
## to install modules, follow instruction from http://www.cpan.org/modules/INSTALL.html
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Cookies;
use IO::Socket::SSL;
use Email::Sender::Simple qw(sendmail);
use Email::Sender::Transport::SMTPS ();
use Email::Simple ();
use Email::Simple::Creator ();
use WWW::Wunderground::API;
use Try::Tiny;
binmode STDOUT, ':utf8'; # for degrees symbol

----
Below is to use Weather Underground API. You must obtain your own free API key to use.

my $weather = new WWW::Wunderground::API(
    location => 'pws:XXXXXXXXXX',
    api_key  => '1111111111111111',
    auto_api => 1,
);

---
In the below code, we paste Message of the Day to our email body and get a list of URLs from a file.

## For debugging purposes
#my $outfile = "/home/me/Documents/perl/output.htm";

## Message of day
my $motd = `exec /usr/games/fortune | /usr/games/cowsay -n`;

## List of websites to check from outside
my $filename = '/home/me/Documents/perl/serverlist.txt';
open(my $fh, '<:encoding(UTF-8)', $filename)
 or die "Could not open file '$filename' $!";

---
Below is the main section used to loop through the list of URLs and check their status using a function call (last block of codes below). In line 10, we also use the WeatherUnderground API to get our temperature.

my $emailbody = "webpage status codes\n";
while (my $text = <$fh>) {
 chomp $text;
 $emailbody = $emailbody  . "$text: ";  
 my $answer = get_answer($text);
  $emailbody = $emailbody  . "$answer\n"; 
}
$emailbody = $emailbody  . "$motd\n"; 

my $currentTemp = $weather->conditions->temp_f;
my $cTempString = "Current Temperature is $currentTemp degrees";
$emailbody = $emailbody  . "$cTempString\n";

##For debugging purposes
#print $emailbody;

---
The below block of code is to connect to my email service provider and send email to everyone in the address book.

## Put SMTP authentication information here
my $smtpserver = 'smtp.office365.com';
my $smtpport = 587;
my $smtpuser   = 'me@outlook.com';
my $smtppassword = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
my $tousers='';
## List of email addresses
$filename = '/home/me/Documents/perl/addresses.txt';
open($fh, '<:encoding(UTF-8)', $filename)
  or die "Could not open file '$filename' $!";

while (my $row = <$fh>) {
   chomp $row;
   $tousers = $tousers  . "$row,";  
}
$tousers = $tousers  . "$smtpuser";

##setup the email server connection here
my $transport = Email::Sender::Transport::SMTPS->new({
   host => $smtpserver,
   port => $smtpport,
   ssl => "starttls",
   sasl_username => $smtpuser,
   sasl_password => $smtppassword,
});

##create email object
my $email = Email::Simple->create(
   header => [
      To      => $tousers,
      From    => $smtpuser,
      Subject => 'Status',
   ],
   body => $emailbody,
);

##send the email
sendmail($email, { transport => $transport });

---
The below code is my function that does the actual website checking. Notice the Try-Catch blocks, this is useful for checking webpage status because if the website is unreachable it is a Catch event. Also line 2 shows how we obtain the parameter value for this function call. Be sure to wrap the variable around ().

sub get_answer {
 my ($row) = @_;
 try{
  chomp $row;
  ##print "$row\n";
      my $mech=WWW::Mechanize->new(ssl_opts => {
         verify_hostname => 0,
      }); 
     $mech->cookie_jar(HTTP::Cookies->new());
     $mech->get($row);
     ##my $output_page = $mech->content();
     my $output_status = $mech->status();
     ##print $output_page; 
  chomp $output_status;
  return $output_status;
 }catch{
    my $output_status = "Fail";
  return $output_status;
 };
}

Wednesday, August 2, 2017

ColdFusion and DataTable Example

If you are new to DataTable like me, this should provide you with the necessary basics to get you going and helps to make sense of examples over at Datables.net.

<html>
<head>
<style>
tfoot input {
      width:100%;
      padding:3px;
      box-sizing:border-box;
}
th,td {
      padding:15px;
}
table#t00 {
      width:100%;
      border-spacing:5px;
      padding:15px;
      border:2px solid #dddddd;
      border-collapse:collapse;
      background-color: #eee;
}
</style>
<link type="text/css" 
      rel="stylesheet" 
      href="https://cdn.datatables.net/1.10.15/css/jquery.dataTables.min.css" 
      media="all"/>
<script type="text/javascript" 
        src="https://cdn.datatables.net/1.10.15/css/jquery.dataTables.min.css">
</script>
</head>

Here's the Head portion of HTML. Line 3-20 is optional, but it is in there so that I can override some style inside linked stylesheet (line 21). Line 12 is there so I can give one table a unique look. Line 21 is one linked stylesheet. You can link as many as you want. Line 25 is datatable JS file. You can also have as many of these as you like.

1
2
3

<cfquery name="somedata" datasource="MYSOURCE">
      select blah1,blah2,blah3 from sometable
</cfquery>

Above is the CFML to call a query.

<body>
  <table id="t00">
    <tr>
      <td>Main Menu</td>
    </tr>
  </table>

Above we have our use of the style we defined earlier for this table Id.

<table id="meatTable" class="table table-striped table-bordered nowrap" border="1">
  <thead>
    <tr>
      <th>Column 1</th>
      <th>Column 2</th>
      <th>Column 3</th>
    </tr>
  </thead>
  <tfoot>
    <tr>
      <th>Column 1</th>
      <th>Column 2</th>
      <th>Column 3</th>
    </tr>
  </tfoot>
  <tbody>
    <cfoutput query="somedata">
      <tr>
        <th>#blah1#</th>
        <th>#blah2#</th>
        <th>#blah3#</th>
      </tr>
    </cfoutput>
  </tbody>
</table>

Above is our main table. We call the CFML in the tbody section to populate it with the previously called query. You need thead section for proper heading behavior (sort) for the table and tfoot section for footer behavior (search). This will become more obvious in the actual javascript portion in the end.

<script type="text/javascript">
  $(document).ready(function(){
    $('#meatTable tfoot th').each( function () {
      var title = $(this).text();
      $(this).html( '<input type="text" placeholder="Search '+title+'" />');
    } );
    var table = $('#meatTable').DataTable();

    table.columns().every( function () {
      var that = this;
      $( 'input', this.footer() ).on( 'keyup change', function () {
        if ( that.search() !== this.value ) {
          that
              .search( this.value )
              .draw();
        }
      } );
    } );
  } );
</script>
</body>
</html>

This is the last portion. We can reference the table that is being used by using single pound sign (line 3 and 7). Line 3 populates the search by column by putting them in the footer of the table. Line 7 is used to declare the Datatable variable which is used in the rest of the script. That is it. Now you should be able to put the pieces together that you find in the DataTables examples page.

Tuesday, August 1, 2017

AWS S3 - Policy to limit access to single bucket

How to limit access to only single S3 bucket

I was recently in a situation where an team wanted to allow someone access to their bucket in our account (from Web Console) but wanted to hide all other buckets. I thought this could be done with some sort of "Deny-All-Except-Condition" policy, but I found that this was harder than anticipated. First of all, I couldn't find a conditional statement that matches against bucket name or bucket tag. There are, however, conditional policy for both prefix string or object tag.

So the below is the best I could do and met the team half way. The below policy permits the user to List all bucket names but they cannot browse into any buckets. And grant user full access to one bucket. The "ListAllMyBuckets" is required, otherwise the user can't use S3 feature from AWS Web Console.

This is a IAM Policy that I attached to the IAM User.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1501602649000",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": [
"*"
]
},
{
"Sid": "Stmt1501602649090",
"Effect": "Allow",
"Action": [
"s3:List*",
"s3:Get*",
"s3:Put*",
"s3:DeleteObject",
"s3:DeleteObjectVersion"
],
"Resource": [
"arn:aws:s3:::myBucketName",
"arn:aws:s3:::myBucketName/*"
]
}
]
}

Please share if you got an easier way to limit exposure of S3 buckets to users.

Friday, July 28, 2017

AWS Cloudwatch - Windows Logs

References:

Basic Steps to get it running (this was tested with EC2 service version 3.19.1153)

Update EC2Config to the latest version
Open EC2ConfigService Settings
Under General Tab, Enable CloudWatch Logs
Copy the sample JSON file to your EC2 install location's settings folder

Download Sample JSON File
c:\program files\amazon\ec2configservice\settings

Edit the JSON file

"Id":"CloudWatchLogs" section should have your information. I leave AccessKey and SecretKey blank because I prefer to use IAM Role that has access to write to CloudWatch. Also, I prefer {hostname} to default, {instance_id} because hostname means something without cross referencing.
"Id":"CloudWatch" section should have your region and NameSpace. NameSpace is the name that you give to your CustomMetrics.

Go to Services, restart "Ec2Config" service.
You should see Application and System Event Logs in your CloudWatch Logs

Configuring Logs in AWS.EC2.Windows.CloudWatch.json file

Windows Logs

Fullname:

AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,

AWS.EC2.Windows.CloudWatch

Id: Update the Id to something unique.

Edit the LogName and Levels to your desired Event and Type of messages. Below are possible values for them.

Possible LogNames (not a complete list). These can be obtained from Windows Event Viewer.

Security
System
Application
Setup
EC2ConfigService
Microsoft-Windows-TerminalServices-LocalSessionManager/Operational

Possible Levels:

1: Error Only
2: Warning Only
4. Information Only
3: Error and Warning
5: Error and Information
6: Warning and Information
7: Error, Warning, and Information

Performance Counters

Fullname:

AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,

AWS.EC2.Windows.CloudWatch

Id: Update the Id to something unique. Example JSON file has "PerformanceCounter." You can use "MemoryCounter" instead. Do not use special characters or spaces in the ID.

CategoryName: These can be obtained from Performance Monitor: Add Counter. Categories are first level values shown on the box on top left. They are shown in blue.

CounterName: These can be obtained by expanding the CategoryName.

InstanceName: These can be obtained from Bottom Left of the Add Counter dialog box. For most this is blank.

MetricName: Some custom metric name that defines this metric

Unit. Possible Values:

DimensionName: Name of the dimension that uniquely identifies this data value. For my situation, I used "ServerName"

DimensionValue: The value for the dimension. For my situation, I used the system variable called, "{hostname}" Other possible values are {instance_id} and {ip_address}, or combination of these three.

Here's my setting:

Custom logs:

Custom logs can be uploaded to cloudwatch provided that it meets certain criteria:

Each entry must begin with the date format following by a space
Log must be one of .NET framework supported text encoding: https://msdn.microsoft.com/en-us/library/system.text.encoding.aspx

Fullname:

AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,

AWS.EC2.Windows.CloudWatch"

LogDirectoryPath: Location of the logs

TimestampFormat: See https://msdn.microsoft.com/en-us/library/8kb3ddd4.aspx

Filter: See https://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.filter.aspx

CultureName: Leave it blank to use local locality settings

TimeZoneKind: Local to use local timezone

LineCount: Number of lines in the header to identity the log file

Custom Metrics not found elsewhere:

If there are metrics that are not mentioned elsewhere, you can push the metrics up via cli (or SDK)

aws cloudwatch put-metric-data 
    --namespace "MyOwnNameSpace"
    --metric-name "Memory_Usage"
    --dimensions "Metric=MegabytesFree,OS=Win,ServerName=MyOwn"
    --unit "Megabytes"
    --value "3000"

Unless timestamp is used, it will upload the metric using current data/time.

Other Logs not mentioned above

IIS Logs: Didn't use it, but it seems pretty simple enough. Just enable it in the Flows to use it as is.
ETW (Event Tracing for Windows): Also didn't use it.

Configuring Flow to post the date in AWS.EC2.Windows.CloudWatch.json file

Find the "Flows" section. Each flow consists of Data ID followed by Destination ID. Using the example JSON file, the two destinations are CloudWatchLogs and CloudWatch. If there are more than one Data ID, then enclose them in (). Here's the example from AWS documentation.