Technical references: August 2017

Wednesday, August 30, 2017

AWS CLI Sample - Find latest snapshot

AWS CLI Sample

Find the most recent snapshots of a volume and update a tag on it

$volumeid = "vol-0000000000000000000"
$output = aws ec2 describe-snapshots `
   --owner-id 4444444444444 `
   --filters "Name=volume-id,Values='$volumeid'" `
   --query 'Snapshots[].{ID:SnapShotId,StartTime:StartTime}'
$output2 = $output | out-string | convertfrom-json
$highDate = "0"
$highID = "0"
foreach($item in $output2){
   if($item.StartTime -gt $highDate){
      $highDate = $item.StartTime
      $highID = $item.ID
   }
}
write-host $highID
$tag_value = "New Stamp (" + $(get-date).GetDateTimeFormats()[93] + ")"
aws ec2 create-tags --resources $highID --tags "Key='Bookmark',Value='$tag_value'"

Lines 2-5 is your standard describe snapshot command. Be sure to include the account number (owner-id) when we're filtering. The query is to limit the data returned since we don't need to know everything about each snapshots.

Lines 9-14 is looping through all the snapshots returned to get the newest one.

Line 16 is just to put a date stamp. GetDateTimeFormats()[93] returns "yyyy-MM-dd hh:mm:ss" date format.

Sunday, August 27, 2017

AWS CloudFormation Sample

AWS CloudFormation - Using Parameters

Parameters are input arguments to CloudFormation script. Items like InterfaceID and InstanceProfile are custom parameters where we define the possible values. NameTag is a customer parameter where input values are open-ended with character limit/type limits. The remainder are AWS Supported Parameter Types, more can be found here.

"Parameters":{
        "ImageId":{
            "Description":"Image ID",
            "Type":"String",
            "Default":"ami-284f1a3e",
            "AllowedValues":[
              "ami-284f1a3e",
              "ami-36174e4d"
            ],
            "ConstraintDescription":""

        },
        "InstanceType":{
            "Description":"Instance Type",
            "Type":"String",
            "Default":"t2.small",
            "AllowedValues":[
       "t2.nano",
       "t2.micro",
       "t2.small",
       "t2.medium",
       "t2.large",
       "t2.xlarge",
       "t2.2xlarge"
            ],
            "ConstraintDescription":""
        },
        "KeyName":{
            "Description":"Key Name",
            "Type":"AWS::EC2::KeyPair::KeyName",
            "ConstraintDescription":""
        },
        "InterfaceID":{
            "Description":"Network Interface ID",
            "Type":"String",
            "Default":"eni-db0e3c7a",
            "AllowedValues":[
              "eni-9158e145",
              "eni-db0e3c7a"
            ],
            "ConstraintDescription":""
        },
        "NameTag":{
            "Description":"Name Tag",
            "Type":"String",
            "Default":"NameTag",
            "MinLength":"1",
            "MaxLength":"10",
            "AllowedPattern":"[a-zA-Z][a-zA-Z0-9]*",
            "ConstraintDescription":""
        },
        "InstanceProfile":{
            "Description":"Instance Profile",
            "Type":"String",
            "Default":"read_only_instance",
            "AllowedValues":[
              "read_only_instance",
              "admin_instance"
            ],
            "ConstraintDescription":""
        }
    }

In the Resources, we reference the above parameters using keyword, "Ref"

"Resources": {
        "MyInstance": {
            "Type": "AWS::EC2::Instance",
            "Properties": {
                "ImageId": {
                    "Ref": "ImageId"
                },
                "InstanceType": {
                    "Ref": "InstanceType"
                },
                "KeyName": {
                    "Ref": "KeyName"
                },
                "NetworkInterfaces": [
                    {
                        "NetworkInterfaceId": 
                            {
                            "Ref": "InterfaceID"
                            },
                        "DeviceIndex":"0"
                    }
                ],
                "Tags": [
                    {
                        "Key": "Name",
                        "Value": {
                            "Ref": "NameTag"
                        }
                    }
                ]
            },
            "Metadata": {
                "AWS::CloudFormation::Designer": {
                    "id": "4b94664e-1e18-400c-9733-ff095ff6e854"
                }
            }
        }
    }

The whole thing looks like this:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters":{
        "ImageId":{
            "Description":"Image ID",
            "Type":"String",
            "Default":"ami-284f1a3e",
            "AllowedValues":[
              "ami-284f1a3e",
              "ami-36174e4d"
            ],
            "ConstraintDescription":""

        },
        "InstanceType":{
            "Description":"Instance Type",
            "Type":"String",
            "Default":"t2.small",
            "AllowedValues":[
       "t2.nano",
       "t2.micro",
       "t2.small",
       "t2.medium",
       "t2.large",
       "t2.xlarge",
       "t2.2xlarge"
            ],
            "ConstraintDescription":""
        },
        "KeyName":{
            "Description":"Key Name",
            "Type":"AWS::EC2::KeyPair::KeyName",
            "ConstraintDescription":""
        },
        "InterfaceID":{
            "Description":"Network Interface ID",
            "Type":"String",
            "Default":"eni-db0e3c7a",
            "AllowedValues":[
              "eni-9158e145",
              "eni-db0e3c7a"
            ],
            "ConstraintDescription":""
        },
        "NameTag":{
            "Description":"Name Tag",
            "Type":"String",
            "Default":"NameTag",
            "MinLength":"1",
            "MaxLength":"10",
            "AllowedPattern":"[a-zA-Z][a-zA-Z0-9]*",
            "ConstraintDescription":""
        },
        "InstanceProfile":{
            "Description":"Instance Profile",
            "Type":"String",
            "Default":"read_only_instance",
            "AllowedValues":[
              "read_only_instance",
              "admin_instance"
            ],
            "ConstraintDescription":""
        }
    },
    "Metadata": {
        "AWS::CloudFormation::Designer": {
            "4b94664e-1e18-400c-9733-ff095ff6e854": {
                "size": {
                    "width": 60,
                    "height": 60
                },
                "position": {
                    "x": 304,
                    "y": 225
                },
                "z": 0
            }
        }
    },
    "Resources": {
        "MyInstance": {
            "Type": "AWS::EC2::Instance",
            "Properties": {
                "ImageId": {
                    "Ref": "ImageId"
                },
                "InstanceType": {
                    "Ref": "InstanceType"
                },
                "KeyName": {
                    "Ref": "KeyName"
                },
                "NetworkInterfaces": [
                    {
                        "NetworkInterfaceId": 
                            {
                            "Ref": "InterfaceID"
                            },
                        "DeviceIndex":"0"
                    }
                ],
                "Tags": [
                    {
                        "Key": "Name",
                        "Value": {
                            "Ref": "NameTag"
                        }
                    }
                ]
            },
            "Metadata": {
                "AWS::CloudFormation::Designer": {
                    "id": "4b94664e-1e18-400c-9733-ff095ff6e854"
                }
            }
        }
    }
}

When you create stack from this:

Thursday, August 10, 2017

Powershell Network Tools

Few Powershell Network tools

This is when you are in an environment that won't let you install software. I've collected these from various sources. I use them often enough to keep it handy here.

Check for TCP Listener:

$socket = new-object net.sockets.tcpclient
$socket.connect("www.google.com",443)
$socket.connected
$socket = $null

Check for TCP Connections on your machine:

This is more than you really need, but it'll print a table of established connections every 5 seconds. If you just want the raw data and all the available connection state, just run lines 2 and 3.

do{
 $tcpproperties=[System.Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties()
 $connections=$tcpproperties.GetActiveTcpConnections()
 $output=@()
 foreach($conn in $connections){
   if($conn.State -eq 'Established'){
     $outputObj = New-Object -typename PSObject
     $outputObj | Add-Member -MemberType NoteProperty -Name "LocalAddress" -Value $conn.LocalEndPoint.Address 
     $outputObj | Add-Member -MemberType NoteProperty -Name "LocalPort" -Value $conn.LocalEndPoint.Port
     $outputObj | Add-Member -MemberType NoteProperty -Name "RemoteAddress" -Value $conn.RemoteEndPoint.Address
     $outputObj | Add-Member -MemberType NoteProperty -Name "RemotePort" -Value $conn.RemoteEndPoint.Port
     $outputObj | Add-Member -MemberType NoteProperty -Name "State" -Value $conn.State
     $output=$output + $outputObj
   }
 }
 $output | format-table
 sleep 5
}while($true)

Check for UDP Listener

For this one, I'm also illustrating a listener too. Good for troubleshooting firewall.

Receiver

Run this portion on your receive end and wait for traffic.

$udpObject = new-object system.net.sockets.udpclient(1433)
$udpObject.client.ReceiveTimeout = 10000
do{
  $remoteEndPoint = new-object System.Net.IPEndPoint([System.Net.IPAddress]::Any,0)
  $receiveBytes = $udpObject.Receive([ref]$remoteEndPoint)
  $a = New-Object System.Text.AsciiEncoding
  [String]$returnData = $a.GetString($receiveBytes)
  Write-host "Received: $($returnData.ToString())"
  Write-host "Sent from: $($remoteEndPoint.Address.ToString()) on their port: $($remoteEndPoint.port.ToString())"
  $ans = read-host "More (y/n)?"
}while($ans -ne 'n')
$udpObject.close()

Sender

Then run this from your remote machine to see the information show up on the remote end.

$udpObject = new-object system.net.sockets.udpclient(1433)
$udpObject.connect("10.10.10.1",1433)
$a = new-object System.Text.AsciiEncoding
$byte = $a.GetBytes("$(get-date)")
[void]$udpObject.send($byte,$byte.length)
$udpObject.close()

Check for UDP Connection

1 2	$udpProperties = [System.Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties() $udpProperties.GetActiveUDPListners()

Remote PowerShell Session

Again, more than you need, but incase you don't want to have to type in your password every time.

$servername = "mydesktop1"
$username = "domain\username"
$password = "password"
$secPassword = convertTo-SecureString $password -AsPlainText -Force
$mycred = New-Object System.Management.Automation.PSCredential($username,$secPassword)
enter-pssession -Credential $mycred -Computername $servername

Send Email

Good way to test your SMTP Server.

$SMTP = "mail.mycompany.net"
$Subject = "Test from $env:Computername"
$toaddress = "myemail@mycompany.net"
$fromaddress = $env:Computername + "@mycompany.net"
$body = "this message was sent via $SMTP at $(get-date)."
send-mailmessage -smtpserver $SMTP -to $toaddress -from $fromaddress -subject $subject -body $body

Wednesday, August 9, 2017

CFML Javascript Update One to Many relationship tables

CFML/Javascript Update Table Page

You have list of users. Those users can belong to 0 to many roles. I needed a simple way to update this table. This is my first approach.

Here's the basic view of the tables and their relationship.

Here's the header portion of the code with the two Javascript functions and the call to my first database table.
alterRole populates a text box that will be submitted when we get ready to save the changes.
resetButton re-enabled all the pressed Add/Delete buttons so that the user can begin again.

<html>
<title>Edit Roles</title>
<head>
<script type ="text/javascript">
function alterRole(action,roleId){
  document.getElementById(roleId).disabled="disabled";
  if(action == 1){
    var txt = document.getElementById("Add").value;
    txt = txt + ";" + roleId;
    document.getElementById("Add").value = txt;
  }else{
    var txt = document.getElementById("Del").value;
    txt = txt + ";" + roleId;
    document.getElementById("Del").value = txt;
  }
}

function resetButton(){
  var inputs = documents.getElementByTagName("BUTTON");
  for (var i=0; i<inputs.length;i++){
    if (inputs[i].type == 'button'){
      input[i].disabled = false;
    }
  }
  document.getElementById("Add").value = "";
  document.getElementById("Del").value = "";
}
</script>

<cfquery name="users" datasource="MYDB">
  select PID, my_UserID from user_list order by userid
</cfquery>
<cfset user_id = 1>
</head>
<cfheader name = "Expires" value "#Now()#">

Below is the first part of the body. This portion is executed when we press Load (user roles) or Save. If we pressed Load then the page loads the currently selected Roles (and available) for the User we selected. If we press Save, then the page inserts into or remove from the intermediate table the UserIDs and RoleIDs. The Form.Add and Form.Del is populated in the last portion of the code.

<body><form action="edit_roles.cfm" method="post">
<cfif isDefined("form.load") or isDefined("form.save")>
  <cfset thisUserID = #form.user_id#>
  <input name="UserName" type="text" value='<cfoutput>#form.user_id#</cfoutput>' size="10" hidden>
  <cfif isDefined("form.save")>
    <cfif len(#form.Add#) gt 0>
      <cfset this1 = #form.Add#>
      <cfset this1 = right(this1,len(this1)-1)>
      <cfset this2 = this1.split(";")>
      <cfloop array = "#this2#" index = "item">
        <cfquery name = "user_to_role" datasource="MYDB">
          insert into user_to_role(user_pid,roleid) values (#thisUserId#,'#item#')
        </cfquery>
      </cfloop>
    </cfif>
    <cfif len(#form.Del#) gt 0>
      <cfset this1 = #form.Del#>
      <cfset this1 = right(this1,len(this1)-1)>
      <cfset this2 = this1.split(";")>
      <cfloop array = "#this2#" index = "item">
        <cfquery name ="user_to_role" datasource="MYDB">
          delete from user_to_role where user_pid= #thisUserId# and roleid = '#item#'
        </cfquery>
      </cfloop>
    </cfif>    
<cfelse>
  <cfset thisUserId = 0>
  <input name ="UserName" type ="text" value ="" size ="10" hidden>
</cfif>

This section populates the entire list of Users and pre-selects the user if we're coming from Save action. This section also has the query that contains the entire list of Roles available and Roles that are already defined for the user.

<h2>User Roles Management</h2>
<h3>Users:
<select name ="user_id" id ="user_id">
  <cfoutput query="users">
    <cfif #PID# eq #thisuserid#>
      <option value="#PID#" selected>#my_USERID#</option>
    <cfelse>
      <option value="#PID#">#my_USERID#</option>
    </cfif>
  </cfoutput>
</selected>
<input name ="Load" type ="Submit" value ="Load"/>
<input name ="Save" type ="Submit" value ="Save"/>
</h3><br>
<cfif #thisuserid# gt 0>
  <cfquery name="user_to_roles" datasource ="MYDB">
    select user_pid, rolename, roles.roleId, ownerid 
    from roles left outer join user_to_role 
    on roles.roleid = user_to_role.roleid 
    and user_pid = #thisuserid# where roletype = 'User' order by rolename
</cfquery>

This section populates a table with the available Roles. If the selected User is already associated with a role then we provide the option to remove, otherwise we have the option add. When we add or delete a role from the selected User, then the RoleID is added to the appropriate text box that will be submitted when we press Save.

<table id ="roles_table">
  <thead>
    <tr>
      <th>RoleName</th>
      <th>Owner</th>
      <th><button type="button" onClick="javascript:resetbutton();">Reset Buttons</button></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <cfoutput query = "user_to_roles">
    <tr>
      <th>#RoleName#</th>
      <th>#Ownerid#</th>
      <cfif len(#user_pid#)>
        <button type="button" onClick="javascript:alterRole(0,'#roleid#');" value ="#roleid#" id = "#roleid#" name = "del_btn">
        Remove</button>
      <cfelse>
        <button type="button" onClick="javascript:alterRole(1,'#roleid#');" value ="#roleid#" id = "#roleid#" name = "add_btn">
        Add</button>
      </cfif>
      <th></th>
      <th></th>
    </tr>
    </cfoutput>
  </tbody>
  </table>
</cfif>
<input name ="Add" id ="Add" type ="text" value ="" size ="100" hidden>
<input name ="Del" id ="Del" type ="text" value ="" size ="100" hidden>
</form>
</body>
</html>

For next version, I plan on using AJAX on every Add/Delete action so that we don't have to submit the entire page. Also, some IF branches can be slimmed down if we move some common actions into a function in CFC.

---

Thursday, August 3, 2017

Perl Webpage status checker

How to check webpage status using Perl

We have several web pages that we needs to be monitored from outside of our work network especially after a maintenance window. I didn't want to do this manually (ever again), so I wrote a Perl script to run on my Linux server (laptop) from home.

This is the header portion. I imported these modules to be used by my script. Line 3 turns any expression that is deemed difficult to debug into error. And line 4 enabled optional warnings.

#!/usr/local/bin/perl
## to install modules, follow instruction from http://www.cpan.org/modules/INSTALL.html
use strict;
use warnings;
use WWW::Mechanize;
use HTTP::Cookies;
use IO::Socket::SSL;
use Email::Sender::Simple qw(sendmail);
use Email::Sender::Transport::SMTPS ();
use Email::Simple ();
use Email::Simple::Creator ();
use WWW::Wunderground::API;
use Try::Tiny;
binmode STDOUT, ':utf8'; # for degrees symbol

----
Below is to use Weather Underground API. You must obtain your own free API key to use.

my $weather = new WWW::Wunderground::API(
    location => 'pws:XXXXXXXXXX',
    api_key  => '1111111111111111',
    auto_api => 1,
);

---
In the below code, we paste Message of the Day to our email body and get a list of URLs from a file.

## For debugging purposes
#my $outfile = "/home/me/Documents/perl/output.htm";

## Message of day
my $motd = `exec /usr/games/fortune | /usr/games/cowsay -n`;

## List of websites to check from outside
my $filename = '/home/me/Documents/perl/serverlist.txt';
open(my $fh, '<:encoding(UTF-8)', $filename)
 or die "Could not open file '$filename' $!";

---
Below is the main section used to loop through the list of URLs and check their status using a function call (last block of codes below). In line 10, we also use the WeatherUnderground API to get our temperature.

my $emailbody = "webpage status codes\n";
while (my $text = <$fh>) {
 chomp $text;
 $emailbody = $emailbody  . "$text: ";  
 my $answer = get_answer($text);
  $emailbody = $emailbody  . "$answer\n"; 
}
$emailbody = $emailbody  . "$motd\n"; 

my $currentTemp = $weather->conditions->temp_f;
my $cTempString = "Current Temperature is $currentTemp degrees";
$emailbody = $emailbody  . "$cTempString\n";

##For debugging purposes
#print $emailbody;

---
The below block of code is to connect to my email service provider and send email to everyone in the address book.

## Put SMTP authentication information here
my $smtpserver = 'smtp.office365.com';
my $smtpport = 587;
my $smtpuser   = 'me@outlook.com';
my $smtppassword = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
my $tousers='';
## List of email addresses
$filename = '/home/me/Documents/perl/addresses.txt';
open($fh, '<:encoding(UTF-8)', $filename)
  or die "Could not open file '$filename' $!";

while (my $row = <$fh>) {
   chomp $row;
   $tousers = $tousers  . "$row,";  
}
$tousers = $tousers  . "$smtpuser";

##setup the email server connection here
my $transport = Email::Sender::Transport::SMTPS->new({
   host => $smtpserver,
   port => $smtpport,
   ssl => "starttls",
   sasl_username => $smtpuser,
   sasl_password => $smtppassword,
});

##create email object
my $email = Email::Simple->create(
   header => [
      To      => $tousers,
      From    => $smtpuser,
      Subject => 'Status',
   ],
   body => $emailbody,
);

##send the email
sendmail($email, { transport => $transport });

---
The below code is my function that does the actual website checking. Notice the Try-Catch blocks, this is useful for checking webpage status because if the website is unreachable it is a Catch event. Also line 2 shows how we obtain the parameter value for this function call. Be sure to wrap the variable around ().

sub get_answer {
 my ($row) = @_;
 try{
  chomp $row;
  ##print "$row\n";
      my $mech=WWW::Mechanize->new(ssl_opts => {
         verify_hostname => 0,
      }); 
     $mech->cookie_jar(HTTP::Cookies->new());
     $mech->get($row);
     ##my $output_page = $mech->content();
     my $output_status = $mech->status();
     ##print $output_page; 
  chomp $output_status;
  return $output_status;
 }catch{
    my $output_status = "Fail";
  return $output_status;
 };
}

Wednesday, August 2, 2017

ColdFusion and DataTable Example

If you are new to DataTable like me, this should provide you with the necessary basics to get you going and helps to make sense of examples over at Datables.net.

<html>
<head>
<style>
tfoot input {
      width:100%;
      padding:3px;
      box-sizing:border-box;
}
th,td {
      padding:15px;
}
table#t00 {
      width:100%;
      border-spacing:5px;
      padding:15px;
      border:2px solid #dddddd;
      border-collapse:collapse;
      background-color: #eee;
}
</style>
<link type="text/css" 
      rel="stylesheet" 
      href="https://cdn.datatables.net/1.10.15/css/jquery.dataTables.min.css" 
      media="all"/>
<script type="text/javascript" 
        src="https://cdn.datatables.net/1.10.15/css/jquery.dataTables.min.css">
</script>
</head>

Here's the Head portion of HTML. Line 3-20 is optional, but it is in there so that I can override some style inside linked stylesheet (line 21). Line 12 is there so I can give one table a unique look. Line 21 is one linked stylesheet. You can link as many as you want. Line 25 is datatable JS file. You can also have as many of these as you like.

1
2
3

<cfquery name="somedata" datasource="MYSOURCE">
      select blah1,blah2,blah3 from sometable
</cfquery>

Above is the CFML to call a query.

<body>
  <table id="t00">
    <tr>
      <td>Main Menu</td>
    </tr>
  </table>

Above we have our use of the style we defined earlier for this table Id.

<table id="meatTable" class="table table-striped table-bordered nowrap" border="1">
  <thead>
    <tr>
      <th>Column 1</th>
      <th>Column 2</th>
      <th>Column 3</th>
    </tr>
  </thead>
  <tfoot>
    <tr>
      <th>Column 1</th>
      <th>Column 2</th>
      <th>Column 3</th>
    </tr>
  </tfoot>
  <tbody>
    <cfoutput query="somedata">
      <tr>
        <th>#blah1#</th>
        <th>#blah2#</th>
        <th>#blah3#</th>
      </tr>
    </cfoutput>
  </tbody>
</table>

Above is our main table. We call the CFML in the tbody section to populate it with the previously called query. You need thead section for proper heading behavior (sort) for the table and tfoot section for footer behavior (search). This will become more obvious in the actual javascript portion in the end.

<script type="text/javascript">
  $(document).ready(function(){
    $('#meatTable tfoot th').each( function () {
      var title = $(this).text();
      $(this).html( '<input type="text" placeholder="Search '+title+'" />');
    } );
    var table = $('#meatTable').DataTable();

    table.columns().every( function () {
      var that = this;
      $( 'input', this.footer() ).on( 'keyup change', function () {
        if ( that.search() !== this.value ) {
          that
              .search( this.value )
              .draw();
        }
      } );
    } );
  } );
</script>
</body>
</html>

This is the last portion. We can reference the table that is being used by using single pound sign (line 3 and 7). Line 3 populates the search by column by putting them in the footer of the table. Line 7 is used to declare the Datatable variable which is used in the rest of the script. That is it. Now you should be able to put the pieces together that you find in the DataTables examples page.

Tuesday, August 1, 2017

AWS S3 - Policy to limit access to single bucket

How to limit access to only single S3 bucket

I was recently in a situation where an team wanted to allow someone access to their bucket in our account (from Web Console) but wanted to hide all other buckets. I thought this could be done with some sort of "Deny-All-Except-Condition" policy, but I found that this was harder than anticipated. First of all, I couldn't find a conditional statement that matches against bucket name or bucket tag. There are, however, conditional policy for both prefix string or object tag.

So the below is the best I could do and met the team half way. The below policy permits the user to List all bucket names but they cannot browse into any buckets. And grant user full access to one bucket. The "ListAllMyBuckets" is required, otherwise the user can't use S3 feature from AWS Web Console.

This is a IAM Policy that I attached to the IAM User.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1501602649000",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": [
"*"
]
},
{
"Sid": "Stmt1501602649090",
"Effect": "Allow",
"Action": [
"s3:List*",
"s3:Get*",
"s3:Put*",
"s3:DeleteObject",
"s3:DeleteObjectVersion"
],
"Resource": [
"arn:aws:s3:::myBucketName",
"arn:aws:s3:::myBucketName/*"
]
}
]
}

Please share if you got an easier way to limit exposure of S3 buckets to users.