Merge pull request #124 from scottslowe/pulumi

Create a Pulumi learning environment

pulumi/README.md

@@ -0,0 +1,63 @@
+# A Pulumi "Sandbox" Environment
+
+This set of files was created to help users establish a "sandbox" environment for playing around with [Pulumi](https://pulumi.io).
+
+**NOTE:** At some point in the future the Ansible playbooks here will be reconciled with the Pulumi role found in the `ansible/pulumi-env` directory of this repository.
+
+## Contents
+
+* **ansible.cfg**: This Ansible configuration file configures Ansible to work with local Vagrant-powered VMs. Modifications are needed to this file if you wish to use Ansible with a different inventory source (like EC2 instances, for example).
+
+* **configure.yml**: This Ansible playbook is used to configure the baseline Pulumi environment. It is written for Ubuntu 18.04, and can be used with either local Vagrant VMs or remote cloud instances (such as EC2 instances). If you wish to use it with the latter, you will need to modify the Ansible configuration file to specify an appropriate inventory source.
+
+* **js-simple-ec2.yml**: This Ansible playbook installs the files necessary to use Pulumi with JavaScript to instantiate an EC2 instance.
+
+* **machines.yml**: This YAML file contains a list of VM definitions and associated configuration data. It is referenced by `Vagrantfile` when Vagrant instantiates the VMs.
+
+* **packer.json**: This Packer build file allows you to create an AWS AMI that is pre-configured with Pulumi.
+
+* **README.md**: This file you're currently reading.
+
+* **roles**: This directory contains all the various Ansible roles for the particular Pulumi example files. Each role directory corresponds to the name of an Ansible playbook, like `js-simple-ec2.yml`.
+
+* **vagrant.py**: This dynamic inventory source for Ansible pulls inventory data from Vagrant. If you wish to use the Ansible playbooks with a different target, you'll need to find an equivalent dynamic inventory source (or create a static inventory) and modify the Ansible configuration file accordingly.
+
+* **Vagrantfile**: This file is used by Vagrant to spin up the virtual machines. This file is fairly extensively commented to help explain what's happening. You should be able to use this file unchanged; all the VM configuration options are stored outside this file.
+
+## Instructions
+
+These instructions assume you've already installed Ansible, your virtualization provider (VMware Fusion/Workstation or VirtualBox), Vagrant, and any necessary plugins (such as the Vagrant VMware plugin). Please refer to the documentation for those products for more information on installation or configuration. Similarly, if you wish to use these Ansible playbooks with a cloud provider such as AWS, these instructions assume all necessary tools and configuration have already been handled.
+
+### Using Vagrant
+
+1. Use `vagrant box add` to add an Ubuntu 18.04 base box for your particular virtualization provider. Some sample boxes are provided in `machines.yml`.
+
+2. If you decided to use a box _other_ than one of the boxes listed here, you'll need to edit `machines.yml` to specify the correct box. In `machines.yml`, the "vmw" line is for a VMware-formatted box, the "vb" line is for the name of a VirtualBox-formatted box, and the "lv" line is for the name of a box that supports the Libvirt provider. Edit the appropriate line based on your virtualization provider and the name of the box you added in step 1.
+
+3. Run `vagrant up` to instantiate the VM.
+
+### Using AWS
+
+1. Download a dynamic inventory script for EC2 (see [here](https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py) for one such script). Perform any necessary configuration to make sure the inventory script is returning the desired results.
+
+2. Modify the `ansible.cfg` to specify the dynamic inventory script downloaded and configured in step 1.
+
+3. Create an EC2 instance using the tooling of your choice (AWS CLI, AWS console, Terraform, CloudFormation, etc.).
+
+Optionally, you can create a preconfigured Pulumi AMI using `packer`:
+
+    packer build packer.json
+
+You'll need to have `packer` installed and configured appropriately.
+
+### For All Platforms
+
+Once you've established a VM or instance using one of the previous two sections, then you can continue here.
+
+1. Run `ansible-playbook configure.yml` to configure the Pulumi base components on the target VM or instance. If you are using a preconfigured AMI built using `packer`, this step isn't needed as it has already been done.
+
+2. Run `ansible-playbook <scenario>.yml` to copy files needed for the particular Pulumi scenario you'd like to use. For example, `js-simple-ec2.yml` is one example scenario that shows using JavaScript with Pulumi to create an EC2 instance.
+
+3. Log into the VM or instance using SSH (for Vagrant use `vagrant ssh`, for example) and switch to the `pulumi` directory in the home directory. The files for the scenario you selected in step 2 will be found there.
+
+4. Play around with Pulumi. Enjoy!

pulumi/Vagrantfile

@@ -0,0 +1,77 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# Specify minimum Vagrant version and Vagrant API version
+Vagrant.require_version '>= 1.6.0'
+VAGRANTFILE_API_VERSION = '2'
+
+# Require 'yaml' module
+require 'yaml'
+
+# Read YAML file with VM details (box, CPU, RAM, IP addresses)
+# Edit machines.yml to change VM configuration details
+machines = YAML.load_file(File.join(File.dirname(__FILE__), 'machines.yml'))
+
+# Create and configure the VMs
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+
+  # Always use Vagrant's default insecure key
+  config.ssh.insert_key = false
+
+  # Iterate through entries in YAML file to create VMs
+  machines.each do |machine|
+
+    # Configure the VMs per details in machines.yml
+    config.vm.define machine['name'] do |srv|
+
+      # Don't check for box updates
+      srv.vm.box_check_update = false
+
+      # Specify the hostname of the VM
+      srv.vm.hostname = machine['name']
+
+      # Specify the Vagrant box to use (use VMware box by default)
+      srv.vm.box = machine['box']['vmw']
+
+      # Configure default synced folder (disable by default)
+      if machine['sync_disabled'] != nil
+        srv.vm.synced_folder '.', '/vagrant', disabled: machine['sync_disabled']
+      else
+        srv.vm.synced_folder '.', '/vagrant', disabled: true
+      end #if machine['sync_disabled']
+
+      # Assign additional private network
+      if machine['ip_addr'] != nil
+        srv.vm.network 'private_network', ip: machine['ip_addr']
+      end # if machine['ip_addr']
+
+      # Configure CPU & RAM per settings in machines.yml (Fusion)
+      srv.vm.provider 'vmware_fusion' do |vmw|
+        vmw.vmx['memsize'] = machine['ram']
+        vmw.vmx['numvcpus'] = machine['vcpu']
+        if machine['nested'] == true
+          vmw.vmx['vhv.enable'] = 'TRUE'
+        end #if machine['nested']
+      end # srv.vm.provider 'vmware_fusion'
+
+      # Configure CPU & RAM per settings in machines.yml (VirtualBox)
+      srv.vm.provider 'virtualbox' do |vb, override|
+        vb.memory = machine['ram']
+        vb.cpus = machine['vcpu']
+        override.vm.box = machine['box']['vb']
+        vb.customize ['modifyvm', :id, '--nictype1', 'virtio']
+        vb.customize ['modifyvm', :id, '--nictype2', 'virtio']
+      end # srv.vm.provider 'virtualbox'
+
+      # Configure CPU & RAM per settings in machines.yml (Libvirt)
+      srv.vm.provider 'libvirt' do |lv,override|
+        lv.memory = machine['ram']
+        lv.cpus = machine['vcpu']
+        override.vm.box = machine['box']['lv']
+        if machine['nested'] == true
+          lv.nested = true
+        end # if machine['nested']
+      end # srv.vm.provider 'libvirt'
+    end # config.vm.define
+  end # machines.each
+end # Vagrant.configure

pulumi/ansible.cfg

@@ -0,0 +1,6 @@
+[defaults]
+inventory = ./vagrant.py
+private_key_file = ~/.vagrant.d/insecure_private_key
+remote_user = vagrant
+host_key_checking = False
+retry_files_enabled = False

pulumi/configure.yml

@@ -0,0 +1,59 @@
+---
+- hosts: "all"
+  become: "yes"
+  remote_user: "vagrant"
+  gather_facts: false
+
+  pre_tasks:
+  - name: "Install Python 2 if not present"
+    raw: "apt -qqy update; apt -qqy install python"
+    register: output
+    retries: 3
+    delay: 3
+    until: output.rc == 0
+  
+  tasks:
+  - name: "Ensure /usr/local/bin exists"
+    file:
+      state: "directory"
+      path: "/usr/local/bin"
+      owner: "root"
+      group: "root"
+  
+  - name: "Check for existence of Pulumi binary in /usr/local/bin"
+    stat:
+      path: "/usr/local/bin/pulumi"
+    register: pulumi_present
+  
+  - name: Install NodeJS
+    apt:
+      state: "present"
+      name: "{{ item }}"
+    with_items:
+      - nodejs
+      - npm
+
+  - block:
+    - name: "Download and extract Pulumi 0.17.5"
+      unarchive:
+        src: "https://get.pulumi.com/releases/sdk/pulumi-v0.17.5-linux-x64.tar.gz"
+        remote_src: true
+        dest: "/tmp"
+        owner: "root"
+        group: "root"
+    - name: "Move binaries to /usr/local/bin"
+      copy:
+        src: "/tmp/pulumi/{{ item }}"
+        remote_src: true
+        dest: "/usr/local/bin"
+        owner: "root"
+        group: "root"
+        mode: 0755
+      with_items:
+        - "pulumi"
+        - "pulumi-language-go"
+        - "pulumi-language-nodejs"
+        - "pulumi-language-python"
+        - "pulumi-language-python-exec"
+        - "pulumi-resource-pulumi-nodejs"
+    when: pulumi_present.stat.exists == False

pulumi/js-simple-ec2.yml

@@ -0,0 +1,5 @@
+---
+- name: Prepare sandbox environment for js-simple-ec2
+  hosts: all
+  roles:
+  - role: js-simple-ec2

pulumi/machines.yml

@@ -0,0 +1,9 @@
+---
+- box:
+    vmw: "generic/ubuntu1804"
+    vb: "ubuntu/bionic64"
+    lv: "generic/ubuntu1804"
+  name: "pulumi"
+  nested: false
+  ram: "1024"
+  vcpu: "1"

pulumi/packer.json

@@ -0,0 +1,34 @@
+{
+    "variables": {
+      "aws_access_key": "",
+      "aws_secret_key": "",
+      "aws_region": "us-west-2",
+      "ubuntu_1804_ami": "ami-03804ed633fe58109"
+    },
+    "builders": [
+      {
+        "name": "ami-pulumi-ubuntu",
+        "type": "amazon-ebs",
+        "instance_type": "t2.small",
+        "source_ami": "{{user `ubuntu_1804_ami`}}",
+        "ami_name": "ami-pulumi-ubuntu-{{timestamp}}",
+        "access_key": "{{user `aws_access_key`}}",
+        "secret_key": "{{user `aws_secret_key`}}",
+        "region": "{{user `aws_region`}}",
+        "ssh_username": "ubuntu",
+        "tags": {
+          "source_ami": "{{user `ubuntu_1804_ami`}}",
+          "build_date": "{{isotime}}",
+          "distribution": "Ubuntu",
+          "distribution_release": "bionic",
+          "distribution_version": "18.04"
+        }
+      }
+    ],
+    "provisioners": [
+      {
+        "type": "ansible",
+        "playbook_file": "./configure.yml"
+      }
+    ]
+  }

pulumi/roles/js-simple-ec2/files/Pulumi.yaml

@@ -0,0 +1,8 @@
+name: ec2instance
+runtime: nodejs
+description: Basic example of an AWS EC2 instance
+template:
+  config:
+    aws:region:
+      description: The AWS region to deploy into
+      default: us-west-2

pulumi/roles/js-simple-ec2/files/index.js

@@ -0,0 +1,39 @@
+"use strict";
+
+const aws = require("@pulumi/aws");
+
+// Specify instance size/type
+let size = "t2.micro";    // t2.micro is available in the AWS free tier
+
+// Specify AMI
+let ami  = "ami-09b42c38b449cfa59"; // AMI for Ubuntu 16.04 in us-west-2 (Oregon)
+
+// Specify key pair to use
+// YOU MUST REPLACE THIS VALUE WITH THE CORRECT NAME FOR YOUR ACCOUNT!
+let keypair = "aws_rsa";
+
+// Create a new security group for port 80
+let group = new aws.ec2.SecurityGroup("pulumi-secgrp", {
+    ingress: [
+        { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
+        { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
+    ],
+});
+
+// (optional) create a simple web server using the startup script for the instance
+let userData =
+`#!/bin/bash
+echo "Hello, World!" > index.html
+nohup python -m SimpleHTTPServer 80 &`;
+
+let server = new aws.ec2.Instance("pulumi-ubuntu", {
+    tags: { "Name": "pulumi-ubuntu" },
+    instanceType: size,
+    securityGroups: [ group.name ], // reference the group object above
+    ami: ami,
+    userData: userData,             // start a simple web server
+    keyName: keypair
+});
+
+exports.publicIp = server.publicIp;
+exports.publicHostName = server.publicDns;

pulumi/roles/js-simple-ec2/files/package.json

@@ -0,0 +1,9 @@
+{
+    "name": "ec2instance",
+    "version": "0.1.0",
+    "main": "index.js",
+    "dependencies": {
+        "@pulumi/pulumi": "latest",
+        "@pulumi/aws": "latest"
+    }
+}

pulumi/roles/js-simple-ec2/tasks/main.yml

@@ -0,0 +1,22 @@
+---
+- name: "Create directory to store files"
+  file:
+    state: "directory"
+    path: "{{ ansible_env.HOME }}/pulumi"
+    owner: "vagrant"
+    group: "vagrant"
+
+- name: "Copy files into sandbox environment"
+  copy:
+    src: "{{ item }}"
+    dest: "{{ ansible_env.HOME }}/pulumi"
+    owner: "vagrant"
+    group: "vagrant"
+  with_items:
+    - "index.js"
+    - "package.json"
+    - "Pulumi.yaml"
+
+- name: "Install Pulumi SDK"
+  npm:
+    path: "{{ ansible_env.HOME }}/pulumi"

pulumi/vagrant.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python
+"""
+Vagrant external inventory script. Automatically finds the IP of the booted vagrant vm(s), and
+returns it under the host group 'vagrant'
+
+Example Vagrant configuration using this script:
+
+    config.vm.provision :ansible do |ansible|
+      ansible.playbook = "./provision/your_playbook.yml"
+      ansible.inventory_file = "./provision/inventory/vagrant.py"
+      ansible.verbose = true
+    end
+"""
+
+# Copyright (C) 2013  Mark Mandel <mark@compoundtheory.com>
+#               2015  Igor Khomyakov <homyakov@gmail.com>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Thanks to the spacewalk.py inventory script for giving me the basic structure
+# of this.
+#
+
+import sys
+import os.path
+import subprocess
+import re
+from paramiko import SSHConfig
+from optparse import OptionParser
+from collections import defaultdict
+import json
+
+from ansible.module_utils._text import to_text
+from ansible.module_utils.six.moves import StringIO
+
+
+_group = 'vagrant'  # a default group
+_ssh_to_ansible = [('user', 'ansible_user'),
+                   ('hostname', 'ansible_host'),
+                   ('identityfile', 'ansible_ssh_private_key_file'),
+                   ('port', 'ansible_port')]
+
+# Options
+# ------------------------------
+
+parser = OptionParser(usage="%prog [options] --list | --host <machine>")
+parser.add_option('--list', default=False, dest="list", action="store_true",
+                  help="Produce a JSON consumable grouping of Vagrant servers for Ansible")
+parser.add_option('--host', default=None, dest="host",
+                  help="Generate additional host specific details for given host for Ansible")
+(options, args) = parser.parse_args()
+
+#
+# helper functions
+#
+
+
+# get all the ssh configs for all boxes in an array of dictionaries.
+def get_ssh_config():
+    return dict((k, get_a_ssh_config(k)) for k in list_running_boxes())
+
+
+# list all the running boxes
+def list_running_boxes():
+
+    output = to_text(subprocess.check_output(["vagrant", "status"]), errors='surrogate_or_strict').split('\n')
+
+    boxes = []
+
+    for line in output:
+        matcher = re.search(r"([^\s]+)[\s]+running \(.+", line)
+        if matcher:
+            boxes.append(matcher.group(1))
+
+    return boxes
+
+
+# get the ssh config for a single box
+def get_a_ssh_config(box_name):
+    """Gives back a map of all the machine's ssh configurations"""
+
+    output = to_text(subprocess.check_output(["vagrant", "ssh-config", box_name]), errors='surrogate_or_strict')
+    config = SSHConfig()
+    config.parse(StringIO(output))
+    host_config = config.lookup(box_name)
+
+    # man 5 ssh_config:
+    # > It is possible to have multiple identity files ...
+    # > all these identities will be tried in sequence.
+    for id in host_config['identityfile']:
+        if os.path.isfile(id):
+            host_config['identityfile'] = id
+
+    return dict((v, host_config[k]) for k, v in _ssh_to_ansible)
+
+
+# List out servers that vagrant has running
+# ------------------------------
+if options.list:
+    ssh_config = get_ssh_config()
+    meta = defaultdict(dict)
+
+    for host in ssh_config:
+        meta['hostvars'][host] = ssh_config[host]
+
+    print(json.dumps({_group: list(ssh_config.keys()), '_meta': meta}))
+    sys.exit(0)
+
+# Get out the host details
+# ------------------------------
+elif options.host:
+    print(json.dumps(get_a_ssh_config(options.host)))
+    sys.exit(0)
+
+# Print out help
+# ------------------------------
+else:
+    parser.print_help()
+    sys.exit(0)