Small Fry

Accessing Powershell class properties within a ScriptBlock

This really drove me nuts for a few minutes.

I had a great set of classes set up and working to do some good stuff. Some of the good stuff included generating ScriptBlock objects to be executed on remote servers via WinRM. Pretty easy so far, but I found some duplication of code blocks in different methods which generated the ScriptBlock's, and I wanted to make use of inheritance to prevent duplication.

 

As it turns out, this is not straightforward at all.

In a "normal" case, reaching outside of the ScriptBlock is accomplished with the majickal $Using variable, which provides references to members directly outside the scope of the block.
For example:

...

[String] $foo = "bar"
[ScriptBlock] $Script =
{
    Write-Host $Using:foo
}
Invoke-Command -ComputerName 'some computer' -Credential $SomeCredential -ScriptBlock $Script
# Echo's bar to the command line

So let's say this is taking place inside an instance of class NoWorky, like this:

[String] $foo
class NoWorky : SomeParentClass
{
    NoWorky([String] $foo : base($foo))
    {
         $this.foo = $foo
    }
    [ScriptBlock] makeFooer()
    {
        [ScriptBlock] $Script =
        {
            Write-Host $this.foo
            # Doesn't work! Script block has no idea about $this!
        }
        return $Script
    }
}

"Okay...", I said to myself. "Perhaps $Using:this.foo?"

[String] $foo
class NoWorky : SomeParentClass
{
    NoWorky([String] $foo : base($foo))
    {
         $this.foo = $foo
    }
    [ScriptBlock] makeFooer()
    {
        [ScriptBlock] $Script =
        {
            Write-Host $Using:this.foo
            # Doesn't work! Script block has no idea about $Using:this!
        }
        return $Script
    }
}

Egad! What is one to do?

 

Enter this crap-hack

So back in the day, before ECMAScript got with the times, you had to do something like this to get the correct "this" to be the "this" in your closures:

var me = this

Yep, that's right. I have to make a local reference of this, and then use that this, by referring to me. Are you throwing up in your mouth a little?


Sooooo, I wound up figuring out that the code needed to look like the following to work:

[String] $foo
class Worky : SomeParentClass
{
    Worky([String] $foo : base($foo))
    {
         $this.foo = $foo
    }
    [ScriptBlock] makeFooer()
    {
        [String] $foo = $this.foo
        [ScriptBlock] $Script =
        {
            Write-Host $Using:foo
            # Works. Outputs bar to console when the returned script block is executed.
        }
        return $Script
    }
}

So that's about it for now. I'm glad I found out how to do this, but I'm still not satisfied. (EVER!)
Next step is to create closures for the class methods, so I can call the methods within the ScriptBlock, without having to copy/paste blocks of code all over the place.

TTFN

The Much Awaited Windows Dockerfile

I have been waiting for Windows containers with Docker ever since I started using Docker period.
What a joy it is, that now I might be able to manage Windows with the same grace and efficiency as Linux all of the sudden?

So here are my initial thoughts, and reactions.

First, there appears to be no support for cross-platform containers as of yet, so Windows containers need to run on WS2016WC or WS2016Nano with Containers support installed via PowerShell.

Very easy to get started once Docker is installed:

docker run microsoft/sample-dotnet

..which produces the following cute ASCIIArt:

  Welcome to .NET Core! __________________ \ \ .... ....' .... .......... .............'..'.. ................'..'..... .......'..........'..'..'.... ........'..........'..'..'..... .'....'..'..........'..'.......'. .'..................'... ...... . ......'......... ..... . ...... .. . .. ...... .... . ....... ...... ....... ............ ................ ...................... ........................'................ ......................'..'...... ....... .........................'..'..... ....... ........ ..'.............'..'.... .......... ..'..'... ...............'....... .......... ...'...... ...... .......... ...... ....... ........... ....... ........ ...... ....... '...'.'. '.'.'.' .... ....... .....'.. ..'..... .. .......... ..'........ ............ .............. ............. '.............. ...........'.. .'.'............ ............... .'.'............. .............'.. ..'..'........... ............... .'.............. ......... .............. .....

Pretty neat.

As cool as that is, I have specific application scopes I'd like to isolate, which is what Docker is really good for; separation of concerns, and codifying infrastructure as code.

So I create a nice simple Dockerfile that looks like this:

FROM microsoft/windowsservercore
ARG PATH
# Lets install Python 2.7
RUN powershell (new-object System.Net.WebClient).Downloadfile('https://www.python.org/ftp/python/2.7.12/python-2.7.12.msi', 'c:\python-2.7.12.msi')
RUN msiexec /qn /i C:\python-2.7.12.msi
RUN del C:\python-2.7.12.msi
ENV Path C:\Python27;C:\Python27\Scripts;$PATH
ENV PYTHONPATH C:\Python27
ENV PYTHONIOENCODING utf-8
RUN assoc .py=Python.File
RUN ftype Python.File=C:\Python27\python.exe "%1" %*

At this point, we can create a new image from this Dockerfile with:

docker build --build-arg PATH=$env:Path -t my_image .

 And run it:

docker run -it my_image powershell
PS C:\

More to come as I go, have a great day.

Adding paths to the PYTHONPATH in a Jenkins Python Build Step

Append a path (Linux specific; use os.pathsep to make it OS agnostic) to $PATH.
I can never remember how to do this, so here is my little reminder.

import os,sys 

cwd=os.getcwd()+"/some_module_directory" 

sys.path+=[cwd]

AWS Crash Course P2

Now for something only slightly different...

So the last time I picked this up, I was manually provisioning instances, and as nice as it is that they've made the UI so user friendly, I'm more interested in creating/configuring/deploying-to instances with code, so I'm going to start looking at the documentation for the Python version of Boto3, which is an SDK for AWS.

My goal for this session is to do the same thing I did last time, but using the SDK to do all the work via a python script.  Should be fun!

Problem 1

In the quick start, and normal API docs for boto3, an access key and secret were mentioned. I was looking all over the place for the access keys I had not yet created, and there wasn't any mention of where to generate the keys other than installing the AWS CLI.  Not that I'm against that, I just want to try to do it all from python.

So I was getting this error at first:

>>> ec2 = boto3.resource('ec2')

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/boto3/__init__.py", line 92, in resource

    return _get_default_session().resource(*args, **kwargs)

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/boto3/__init__.py", line 72, in _get_default_session

    setup_default_session()

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/boto3/__init__.py", line 34, in setup_default_session

    DEFAULT_SESSION = Session(**kwargs)

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/boto3/session.py", line 80, in __init__

    self._setup_loader()

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/boto3/session.py", line 113, in _setup_loader

    self._loader = self._session.get_component('data_loader')

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/session.py", line 691, in get_component

    return self._components.get_component(name)

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/session.py", line 885, in get_component

    self._components[name] = factory()

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/session.py", line 179, in <lambda>

    lambda:  create_loader(self.get_config_variable('data_path')))

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/session.py", line 259, in get_config_variable

    elif self._found_in_config_file(methods, var_config):

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/session.py", line 280, in _found_in_config_file

    return var_config[0] in self.get_scoped_config()

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/session.py", line 342, in get_scoped_config

    profile_map = self._build_profile_map()

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/session.py", line 214, in _build_profile_map

    self._profile_map = self.full_config['profiles']

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/session.py", line 380, in full_config

    cred_file)

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/configloader.py", line 131, in raw_config_parse

    raise botocore.exceptions.ConfigParseError(path=path)

botocore.exceptions.ConfigParseError: Unable to parse config file:

...which really makes sense now, since my config was mangel-wurzel at this point.

So I found the Access Key widget in the AWS account info/management for my account.  Problem solved:

>>> ec2 = boto3.resource('ec2')

>>> dir(ec2)

[u'ClassicAddress', u'DhcpOptions', u'Image', u'Instance', u'InternetGateway', u'KeyPair', u'NetworkAcl', u'NetworkInterface', u'NetworkInterfaceAssociation', u'PlacementGroup', u'Route', u'RouteTable', u'RouteTableAssociation', u'SecurityGroup', u'Snapshot', u'Subnet', u'Tag', u'Volume', u'Vpc', u'VpcAddress', u'VpcPeeringConnection', '__class__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', u'classic_addresses', u'create_dhcp_options', u'create_instances', u'create_internet_gateway', u'create_key_pair', u'create_network_acl', u'create_network_interface', u'create_placement_group', u'create_route_table', u'create_security_group', u'create_snapshot', u'create_subnet', u'create_tags', u'create_volume', u'create_vpc', u'create_vpc_peering_connection', u'dhcp_options_sets', u'disassociate_route_table', 'get_available_subresources', u'images', u'import_key_pair', u'instances', u'internet_gateways', u'key_pairs', 'meta', u'network_acls', u'network_interfaces', u'placement_groups', u'register_image', u'route_tables',  u'security_groups', u'snapshots', u'subnets', u'volumes', u'vpc_addresses', u'vpc_peering_connections', u'vpcs']

So now lets take a look at the instances I already have:

>>> instances = ec2.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])

>>> instances

ec2.instancesCollection(ec2.ServiceResource(), ec2.Instance)

>>> for i in instances:

...     print(i.id, i.instance_type)

... 

('i-011b26ac17c23c5ac', 't2.micro')

This appears to be the instance I created manually.  Cool beans, how about the instance(s) status?

>>> for status in ec2.meta.client.describe_instance_status()['InstanceStatuses']:

...     print(status)

... 

{u'InstanceId': 'i-011b26ac17c23c5ac', u'InstanceState': {u'Code': 16, u'Name': 'running'}, u'AvailabilityZone': 'us-west-2b', u'SystemStatus': {u'Status': 'ok', u'Details': [{u'Status': 'passed', u'Name': 'reachability'}]}, u'InstanceStatus': {u'Status': 'ok', u'Details': [{u'Status': 'passed', u'Name': 'reachability'}]}}

Cool, now I want to create another instance using the same AMI I used to create the last instance:

>>> ec2.create_instances(ImageId='ami-00bb6f60', MinCount=1, MaxCount=1)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/boto3/resources/factory.py", line 520, in do_action

    response = action(self, *args, **kwargs)

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/boto3/resources/action.py", line 83, in __call__

    response = getattr(parent.meta.client, operation_name)(**params)

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 159, in _api_call

    return self._make_api_call(operation_name, kwargs)

  File "/Users/adam/Library/Python/2.7/lib/python/site-packages/botocore/client.py", line 494, in _make_api_call

    raise ClientError(parsed_response, operation_name)

botocore.exceptions.ClientError: An error occurred (InvalidParameterCombination) when calling the RunInstances operation: Non-Windows instances with a virtualization type of 'hvm' are currently not supported for this instance type.

Hrm, that isn't a pretty exception, but it does give me the info I need to move along. Using a different API which allows for PV (found in public images section of AWS management console)

>>> ec2.create_instances(ImageId='ami-01940631', MinCount=1, MaxCount=1)

[ec2.Instance(id='i-005edb19b06e5f76c')]

...and boom: I've got another instance now:

>>> for status in ec2.meta.client.describe_instance_status()['InstanceStatuses']:

...     print(status)

... 

{u'InstanceId': 'i-011b26ac17c23c5ac', u'InstanceState': {u'Code': 16, u'Name': 'running'}, u'AvailabilityZone': 'us-west-2b', u'SystemStatus': {u'Status': 'ok', u'Details': [{u'Status': 'passed', u'Name': 'reachability'}]}, u'InstanceStatus': {u'Status': 'ok', u'Details': [{u'Status': 'passed', u'Name': 'reachability'}]}}

{u'InstanceId': 'i-005edb19b06e5f76c', u'InstanceState': {u'Code': 16, u'Name': 'running'}, u'AvailabilityZone': 'us-west-2b', u'SystemStatus': {u'Status': 'initializing', u'Details': [{u'Status': 'initializing', u'Name': 'reachability'}]}, u'InstanceStatus': {u'Status': 'initializing', u'Details': [{u'Status': 'initializing', u'Name': 'reachability'}]}}

On to adjusting the states, snapshots, and instance deletion.

AWS Crash Course

It always starts this way...

I have been interviewing a lot lately, and one of the big things I've been lacking in my tool belt has been actual AWS experience.  My go-to response has been to point out that AWS is only a sort of VMWare for the cloud, and that given a short amount of time with some API documentation, and I'd be fine learning on my feet, etc.

I suppose you can't blame them for not having faith in my abilities to learn on my feet.  :)

So at this point, my intention is to document the trials and tribulations I encounter, cracking the AWS box open for the first time ever, with the intention of teaching myself the essentials, and perhaps helping anyone else who might be in need of this.

My gut tells me that it is going to be REALLY EASY, and that there will very likely be numerous SDK's out there that make the management of my little cloud via code even easier.

Let's see how this pans out.

First Impressions

Very easy to create a new account. It appears to be free, though that remains to be seen. 

I can already see that what AWS has to offer is pretty neat. 

It appears that the first thing I'm going to want to do is to launch an EC2 instance.

EC2

Since my goal is to begin at the beginning...

EC2 is the first service type I'm going to delve into.  It seems to be like a VM hosted "out there", so lets see if that hunch turns out to be correct.

Launch EC2 Instance

Wow, they really made this front-end easy to deal with. Not at all difficult to see that I can launch via a prefab AMI (Amazon Machine Image) of RHEL, SLES, and Ubuntu varieties, plus Amazon Linux (probably some CoreOS variant), and of course, Windows.

So I'm looking at this, and assuming that there is a direct analog through an API to do this very same selection of their canned variants, and I will likely be able to create my own variant via Dockerfile/etc.

I'm selecting an Ubuntu image, and (filtering by free tier eligibility) can see that they have created instance types suited for various different use-cases. 

Connecting To My Instance

A quick adjustment to the firewall rules to only allow my IP to the instance, and I'm connected.
They made it VERY EASY to connect to my instance.  A nice little helpful context-sensitive popup gave me exact connection instructions, specific to my instance.

Time to deploy an app.
I'll start manually, then ramp up to deployments via Chef, Docker, etc.

Manual Deployment

Easy first deploy:
    scp -i my.pem source.zip youser@myinstance.com:~/

I kick off my app on the instance, and am immediately able to use it, and with very little latency.
Since this is essentially a VM, I can interact with it in exactly the same manner, so CM with Puppet/Chef/Ansible//Salt/etc, could probably even install the Docker daemon and spawn as many containers I desire.

Next stop, digging up API documentation for AWS so I can do what I just did via python instead of mouse-clicks.

Running 32-bit Docker container on 64-bit host

Our 32-bit LXC containers all run cov-build flawlessly, so I was surprised to find that it wouldn't run on my 32-bit Docker containers unless I preface the command with /usr/bin/linux32, like:

time /usr/bin/linux32 cov-build --dir someDir/coverity make -j16