Publishing Powershell Nuget module to Azure DevOps

Publishing your powershell nuget module to your Azure DevOps Artifacts

Log into your Azure DevOps Project

Click Artifacts

Create Feed

Give it a name, visibility, and Scope

After your feed is created, click Connect to feed and click NuGet to get the package source (you can get this from any type). It'll look like this:

https://pkgs.dev.azure.com/ORG/PROJECT/_packaging/FEED/nuget/v3/index.json

Click on your settings icon and select Personal Access Tokens

Create a new token and give it at least 

• Work Items (Read)
• Packaging (Read & Write)

Save this token for later use.

Create a new working directory for your module

mkdir c:\mud

Go into this directory.

Create a new manifest file here.

New-ModuleManifest -Path .\mud.psd1

Create a new module root file.

New-item .\mud.psm1

Update the new manifest file with at least following

RootModule = 'mud.psm1'
ModuleVersion = '1.1.0'
FunctionsToExport = @('function1','function2')
FileList = @('file1.ps1','file2.ps1')

• Functions are list of functions that you want exposed from this module.
• Files are list of files that will be consumed by this module. These files must reside at the root of this folder. 

Create a nuget spec file using the name of your module (be sure you have downloaded nuget cli)

nuget spec mud

Go into this new file (mud.nuspec)

• Be sure the version matches (with mud.psd1)
• Be sure to update ProjectURL and IconURL or just remove it altogether
• Remove the default dependency

Package your module

nuget pack mud.nuspec

Add the source

nuget sources Add -Name "myfeed" -Source $source_push -username $username -password $pat

• source_push is your package source address from above
• username is probably your email address
• pat is your Personal Access Token you got earlier

Push this package

nuget push -Source "myfeed" -ApiKey AzureDevOpsServices .\mud.nupkg

That's it. Now you can browse, find, and install the above module. 

> Register-PSRepository -Name $repo_name -SourceLocation $source_repo -InstallationPolicy Trusted
> Get-PSRepository
> find-module -Repository $repo_name -credential $credsAzureDevopsServices
> Install-Module -Name $module_name -Repository $repo_name -credential $credsAzureDevopsServices

GitHub and Terraform Cloud

Getting started with Terraform Cloud and GitHub

Creating a new workspace (linked to GitHub)

Click Workspaces

Click New workspace

Choose GitHub.com

Click on the provided link

Choose a repository. You can choose any repo. 

Click Create workspace

Setting Auto Apply

Go to your workspace

Click on Settings >> General

Now when you update a file in the above GitHub repo that you associated with this workspace, it'll cause an Apply to occur. 

Creating a private module

From GitHub, create a new repository. Terraform Cloud only supports 1 module per reository. They MUST be named in following format "terraform-<provider>-<unique name>" In my example, this will be called "terraform-aws-testmodule" because I am creating an aws module.

After you added all your files, you must create a Release and Tag it in this format "v#.#.#" 

Next, go to your Terraform Cloud account

Click Module

Click Add module

Choose GitHub.com 

Click on the link provided and enter the information shown in your GitHub account

Select your repository from the list

If you didn't name the repo correctly, you won't see it. If you didn't tag it properly, you'll also get an error. 

If it worked, you'll see a new module following the version number you tagged and a provision instructions. 

Go back to GitHub and make a second release with new tag. Then go back to Terraform Cloud and refresh the module page from above. 

Using a private module

Go to your code that is associated with your Terraform Cloud workspace. Update the code to include the provision block provided from above:

module "testmodule" {
  source  = "app.terraform.io/BLAH/testmodue/aws"
  version = "2.0.0"
}

In my workspace, this auto-runs on commit.

Using Terraform Cloud API (Python)

Create a separate virtual environment for your Terraform Cloud code (recommended).

Create venv

> py -m venv tfc_env

Activate it

\tfc_env\Scripts\activate
Deactivate it

> deactivate

Install this 

https://github.com/adeo/iwc-tfc-client

> pip install tfc_client --trusted-host pypi.org --trusted-host files.pythonhosted.org

There are other options as well: 

https://www.terraform.io/docs/cloud/api/index.html

Go to Terraform Cloud and retrieve a Token under User Settings

Here's a sample code to interact with Terraform Cloud (more example at above links)

## You need to activate this virtual env before you run this:
##> .\python\Scripts\activate
## When you are done, you should deactivate it:
##> deactivate
import os
## Doc for this is here https://github.com/adeo/iwc-tfc-client
## pip install tfc_client --trusted-host pypi.org --trusted-host files.pythonhosted.org
from tfc_client import TFCClient
from tfc_client.enums import (
    RunStatus,
    NotificationTrigger,
    NotificationsDestinationType,
)
from tfc_client.models import VCSRepoModel

##Needed this Self-Signed Cert when working on VPN
os.environ["REQUESTS_CA_BUNDLE"]="./mycert.pem"
#$env:REQUESTS_CA_BUNDLE="./mycert.pem"
# Instanciate the client
## Get the token from web console and paste it into the file
token = open("token.txt", "r").read()
client = TFCClient(token=token)

# Retreive any object type by ID from the client
my_org = client.get("organization", id="xxxxxxxx")
my_ws_byID = client.get("workspace", id="ws-111111111")
my_ws_byName = my_org.workspace(name="7777777777")
my_run = client.get("run", id="run-777777777777777")
my_var = client.get("var", id="test")

# To retreive all workspaces:
for ws in my_org.workspaces:
    print(ws.name)

print(my_run)
print(my_var)
#my_run = my_ws_byName.create("run", message="Run run run")

If you need full access to their available API, you should consider using their tfe-go library. 

https://www.terraform.io/docs/cloud/api/index.html

Terraform Subnet Splitting

How to dynamically split subnets in Terraform

Using cidrsubnets: 

The one without "s" returns a single subnet whereas the other returns a set of subnets. The "newbits" are used to determine how many additional bits of netmask to use in creating subsequent subnets. 

For example: 10.1.0.0/16. First two octets are not important. To simplify explanation, I'm going to use binary notation of last 2 octets. Here we're asking for 2 subnets with each having one more netmask. 

 

 

output "example"{
    value = cidrsubnets("10.1.0.0/16",1,1)
}

Result:

example = [
  "10.1.0.0/17",
  "10.1.128.0/17",
]

This is allowed because we can have 2 additional subnets with that netmask

  0.0 to 127.255    <- possible range
  00000000.00000000 <- network
  10000000.00000000 <- mask
  
  128.0 to 128.255
  10000000.00000000
  10000000.00000000

If you try to get another subnet, you'll get an error:

Error: Invalid function argument

  on main.tf line 50, in output "example":
  50:     value = cidrsubnets("10.1.0.0/16",1,1,1)

Invalid value for "newbits" parameter: not enough remaining address space for
a subnet with a prefix of 17 bits after 10.1.128.0/17.

Following the above logic, you can see that you can use 2 "newbits" to create 4 subnets.

output "example"{
    value = cidrsubnets("10.1.0.0/16",2,2,2,2)
}

example = [
  "10.1.0.0/18",
  "10.1.64.0/18",
  "10.1.128.0/18",
  "10.1.192.0/18",
]

==========================
  0.0 to 63.255  
  00000000.00000000
  11000000.00000000
  
  64.0 to 127.255
  01000000.00000000
  11000000.00000000
  
  128.0 to 191.255
  10000000.00000000
  11000000.00000000
  
  192.0 to 255.255
  11000000.00000000
  11000000.00000000

And so, using log base 2 to the desired number of subnets, you can calculate how many minimum "newbits" you need to use. 

Using cidrsubnet

However, a big issue here is calling cidrsubnets won't allow you to pass in dynamic number of arguments to create dynamic number of subnets. That's when you can use cidrsubnet and pass in the nth subnets.

This gives us the 4th subnet (it's zero based index):

output "example"{
    value = cidrsubnet("10.1.0.0/16",2,3)
}

example = 10.1.192.0/18

But of course tricky part of Terraform is that there isn't a convenient way to do an index for loop like this:

for (i=0; i<5; i++){

}

Index Looping

You can hack your way around this by creating a map or list and doing a for loop across it. 

Method 1: Pre-populated map of counter array. This method will use the desired index that matches the key in the counter_map. 

locals{
    counter_map={
        1=[0],
        2=[0,1],
        3=[0,1,2],
        4=[0,1,2,3],
        5=[0,1,2,3,4]
    }
    private_subnets = [for item in local.counter_map[var.private_count]: cidrsubnet(local.subnets[1], local.private_count_newbit, item)]
}

Method 2: One long counter array. This method pulls all the numbers up to the desired index value. 

locals{
    counter_set_all = [0,1,2,3,4,5,6,7,8,9]
    public_subnets = [for item in [for item in local.counter_set_all : item  if item < var.public_count]: cidrsubnet(local.subnets[0], local.public_count_newbit, item)]}

Putting all the pieces together

variable "public_count_weight"{
    default = 1
    description = "newbit weight given to subnet, lower means bigger subnet"
} 
variable "private_count_weight"{
    default = 1
    description = "newbit weight given to subnet, lower means bigger subnet"
}
    
variable "private_count"{
    default = 2
}

variable "public_count"{
    default = 2
}

locals{
    ## newbits can be calculated from the number of subnets desired by looking at the binary log to the desired subnet
    ## If 2 subnets are required, this requires 1 more bit in the netmask  (2^1 = 2 or log base 2 of 2 = 1)
    ## If 4 subnets are required, this requires 2 more bits in the netmask (2^2 = 4 or log base 2 of 4 = 2) 
    ## If 8 subnets are required, this requires 3 more bits in the netmask (2^3 = 8 or log base 2 of 8 = 3)
    public_count_newbit  = ceil(log(var.public_count  , 2 ))
    private_count_newbit = ceil(log(var.private_count , 2 ))
    ## split the initial CIDR into two, one for public and one for private
    subnets = cidrsubnets("10.1.0.0/16", var.public_count_weight, var.private_count_weight)
    ## Split each of the half from above for desired number of subnet in each type
    public_subnets = var.public_count == 1 ? [local.subnets[0]]:[for item in [for item in local.counter_set_all : item  if item < var.public_count]: cidrsubnet(local.subnets[0], local.public_count_newbit, item)]
    private_subnets = var.private_count == 1 ? [local.subnets[1]]:[for item in local.counter_map[var.private_count]: cidrsubnet(local.subnets[1], local.private_count_newbit, item)]
}

locals{
    counter_map={
        1=[0],
        2=[0,1],
        3=[0,1,2],
        4=[0,1,2,3],
        5=[0,1,2,3,4]
    }
    counter_set_all = [0,1,2,3,4,5,6,7,8,9]

}

output "subnets" {
    value = local.subnets
}

output "private"{
    value = local.private_subnets
}

output "public"{
    value = local.public_subnets
}