Initial commit for Consul-Docker-Registrator environment

Add initial files and support for Vagrant environment containing
Consul, Docker, and Registrator.

consul-docker-registrator/README.md

@@ -0,0 +1,57 @@
+# Running a Consul Cluster in Vagrant
+
+These files were created to allow users to use Vagrant ([http://www.vagrantup.com](http://www.vagrantup.com)) quickly and relatively easily spin up a Consul ([http://www.consul.io](http://www.consul.io)) cluster. The configuration was tested using Vagrant 1.7.2, VMware Fusion 6.0.5, and the Vagrant VMware plugin.
+
+## Contents
+
+* **boostrap.json**: This Consul configuration file contains configuration directives to bootstrap the Consul cluster. This file is copied to `/etc/consul.d/bootstrap/config.json` by the Vagrant file provisioner. You should not need to edit this file.
+
+* **consul.sh**: This shell script is executed by the Vagrant shell provisioner to provision the Ubuntu base box with the Consul binary. This shell script was written for an Ubuntu system; edits will likely be necessary for use with a different Linux distribution.
+
+* **README.md**: This file you're currently reading.
+
+* **server.json**: This Consul configuration file contains configuration directives to run the Consul agent as a server. This file is copied to `/etc/consul.d/server/config.json` by the Vagrant file provisioner. The IP addresses specified in this file _must_ match the IP address specified in `servers.yml`.
+
+* **servers.yml**: This YAML file contains a list of VM definitions. It is referenced by `Vagrantfile` when Vagrant instantiates the VMs. You will need to edit this file to provide appropriate IP addresses and other VM configuration data (see "Instructions" below). If you edit the IP addresses in this file, you _must_ also edit `server.json` to supply matching IP addresses there as well.
+
+* **Vagrantfile**: This file is used by Vagrant to spin up the virtual machines. This file is fairly extensively commented to help explain what's happening. You should be able to use this file unchanged; all the VM configuration options are stored outside this file.
+
+## Instructions
+
+These instructions assume you've already installed VMware Fusion, Vagrant, and the Vagrant VMware plugin. Please refer to the documentation for those products for more information on installation or configuration.
+
+1. Use `vagrant box add` to install an Ubuntu 14.04 x64 box for the vmware_fusion provider. I have a base box you can use for this purpose; to use my Ubuntu 14.04 x64 base box, add the box with `vagrant box add slowe/ubuntu-trusty-x64`.
+
+2. Place the files from the `consul` directory of this GitHub repository into a directory on your local system. You can clone the entire "learning-tools" repository (using `git clone`) or just download the specific files from the the `consul` folder.
+
+3. Edit the `servers.yml` file to provide the specific details on the VMs that Vagrant should create. The `Vagrantfile` expects five values for each VM: `name` (the user-friendly name of the VM, which will also be used as the hostname for the guest OS inside the VM); `box` (the name of an Ubuntu 14.04 base box); `ram` (the amount of memory to be assigned to the VM); `vcpu` (the number of vCPUs that should be assigned to the VM); and `priv_ip` (an IP address to be statically assigned to the VM and is used for Consul cluster communications).
+
+4. Once you have edited `servers.yml` (and `server.json`, if you changed the IP addresses in `servers.yml`), use `vagrant up` to bring up the 3 systems that will serve as your Consul cluster.
+
+5. Once Vagrant has finished bringing up the VMs, simply use `vagrant ssh consul-01` (where `consul-01` is the value assigned to the first VM from `servers.yml`) to connect to the VM. No password should be required; it should use the default (insecure) SSH key. Once you are logged into the first VM, bootstrap the Consul cluster with this command:
+
+		consul agent -config-dir /etc/consul.d/bootstrap -advertise 192.168.1.101 -client 0.0.0.0
+
+	If you changed the IP address assigned to the first VM in `servers.yml`, then substitute the appropriate address for the `-advertise` parameter in the above command.
+
+6. In a separate terminal window, connect to the second VM using `vagrant ssh consul-02`. (If you changed the name in `servers.yml`, specify the changed name here.) Launch the second member of the Consul cluster with this command:
+
+		consul agent -config-dir /etc/consul.d/server -advertise 192.168.1.102 -client 0.0.0.0
+
+	Change `192.168.1.102` in the command above if you changed the IP address specified for the second VM in `servers.yml`.
+
+7. In yet another terminal window, connect to the third VM using `vagrant ssh consul-03` (or the name provided in `servers.yml` for the third VM). Launch the third member of the Consul cluster with this command (change the IP address to match what is provided in `servers.yml`):
+
+		consul agent -config-dir /etc/consul.d/server -advertise 192.168.1.103 -client 0.0.0.0
+
+8. At this point, return to the first terminal window (where you are connected to the initial Consul instance launched to bootstrap the cluster) and press Ctrl+C to kill that instance. Launch it again as a "normal" Consul instance with this command:
+
+		consul agent -config-dir /etc/consul.d/server -advertise 192.168.1.101 -client 0.0.0.0 -rejoin
+
+	Consul should re-join the other two nodes in the cluster.
+
+At this point, you have a functional Consul cluster running under Vagrant. If you are using VMware Fusion, you should have IP connectivity to the VMs, and can use the OS X `consul` binary to connect to the cluster and test it. For example, this command would work to demonstrate that Consul is working (you would need to change the IP address provided after `-rpc-addr`):
+
+	consul members -rpc-addr=192.168.1.101:8400
+
+Enjoy!

consul-docker-registrator/Vagrantfile

@@ -0,0 +1,61 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# Specify Vagrant version and Vagrant API version
+Vagrant.require_version ">= 1.6.0"
+VAGRANTFILE_API_VERSION = "2"
+
+# Require 'yaml' and 'fileutils' modules
+require 'yaml'
+require 'fileutils'
+
+# Look for user-data file to configure/customize CoreOS boxes
+# No changes should need to be made to this file
+USER_DATA = File.join(File.dirname(__FILE__), "user-data")
+
+# Read YAML file with VM details (box, CPU, RAM, IP addresses)
+# Be sure to edit servers.yml to provide correct IP addresses
+servers = YAML.load_file('servers.yml')
+
+# Create and configure the VMs
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+
+  # Always use Vagrant's default insecure key
+  config.ssh.insert_key = false
+
+  # Iterate through entries in YAML file to create VMs
+  servers.each do |servers|
+    config.vm.define servers["name"] do |srv|
+      # Don't check for box updates
+      srv.vm.box_check_update = false
+      srv.vm.hostname = servers["name"]
+      srv.vm.box = servers["box"]
+      # Assign an additional static private network
+      srv.vm.network "private_network", ip: servers["priv_ip"]
+      # Configure VMs based on CoreOS box
+      if srv.vm.box == "coreos-stable"
+        # Disable default synced folder for CoreOS VMs
+        srv.vm.synced_folder ".", "/vagrant", disabled: true
+        # Copy user_data file into CoreOS VM
+        srv.vm.provision "file", source: "#{USER_DATA}", destination: "/tmp/vagrantfile-user-data"
+        # Move user_data to correct location to be processed by cloud-init
+        srv.vm.provision "shell", inline: "mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/", privileged: true
+      end
+      # Configure VMs based on Ubuntu box
+      if srv.vm.box == "slowe/ubuntu-trusty-x64"
+        # Enable default synced folder
+        srv.vm.synced_folder ".", "/vagrant"
+        # Copy files into the VM
+        srv.vm.provision "file", source: "server.json", destination: "/home/vagrant/config.json"
+        srv.vm.provision "file", source: "consul.conf", destination: "/home/vagrant/consul.conf"
+        # Run final provisioning script
+        srv.vm.provision "shell", path: "consul.sh"
+      end
+      # Configure VMs with RAM and CPUs per settings in servers.yml
+      srv.vm.provider :vmware_fusion do |vmw|
+        vmw.vmx["memsize"] = servers["ram"]
+        vmw.vmx["numvcpus"] = servers["vcpu"]
+      end
+    end
+  end
+end

consul-docker-registrator/consul.conf

@@ -0,0 +1,19 @@
+description "Consul server process"
+
+start on runlevel [2345]
+stop on runlevel [!2345]
+
+respawn
+
+script
+  if [ -f "/etc/service/consul" ]; then
+    . /etc/service/consul
+  fi
+
+export GOMAXPROCS=`nproc`
+
+exec /usr/local/bin/consul agent \
+  -config-dir="/etc/consul.d/server" \
+  ${CONSUL_FLAGS} \
+  >>/var/log/consul.log 2>&1
+end script

consul-docker-registrator/consul.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+
+# Update list of packages
+sudo apt-get update
+
+# Install unzip package needed to decompress Consul download
+export DEBIAN_FRONTEND=noninteractive
+sudo apt-get -y install unzip
+
+# Get Consul download
+wget https://dl.bintray.com/mitchellh/consul/0.5.0_linux_amd64.zip
+
+# Decompress and remove Consul download
+unzip 0.5.0_linux_amd64.zip
+rm 0.5.0_linux_amd64.zip
+
+# Move Consul binary to location in path
+sudo mv consul /usr/local/bin/
+
+# Create consul user
+useradd -M -d /var/consul -r -s /usr/bin/nologin consul
+
+# Create directories needed by Consul
+sudo mkdir /var/consul
+sudo chown -R consul:consul /var/consul
+sudo mkdir -p /etc/consul.d/server
+sudo chown -R root:consul /etc/consul.d
+
+# Move files into the correct locations
+sudo mv /home/vagrant/consul.conf /etc/init/consul.conf
+sudo chown root:root /etc/init/consul.conf
+sudo mv /home/vagrant/config.json /etc/consul.d/server/config.json
+sudo chown root:consul /etc/consul.d/server/config.json

consul-docker-registrator/server.json

@@ -0,0 +1,10 @@
+{
+	"bootstrap_expect": 3,
+	"server": true,
+	"datacenter": "den1",
+	"data_dir": "/var/consul",
+	"log_level": "INFO",
+	"enable_syslog": false,
+	"retry_join": ["192.168.1.101", "192.168.1.102", "192.168.1.103"],
+	"client_addr": "0.0.0.0"
+}

consul-docker-registrator/servers.yml

@@ -0,0 +1,31 @@
+---
+- name: consul-01
+  box: slowe/ubuntu-trusty-x64
+  ram: 512
+  vcpu: 1
+  priv_ip: 192.168.1.101
+- name: consul-02
+  box: slowe/ubuntu-trusty-x64
+  ram: 512
+  vcpu: 1
+  priv_ip: 192.168.1.102
+- name: consul-03
+  box: slowe/ubuntu-trusty-x64
+  ram: 512
+  vcpu: 1
+  priv_ip: 192.168.1.103
+- name: coreos-01
+  box: coreos-stable
+  ram: 512
+  vcpu: 1
+  priv_ip: 192.168.1.104
+- name: coreos-02
+  box: coreos-stable
+  ram: 512
+  vcpu: 1
+  priv_ip: 192.168.1.105
+- name: coreos-03
+  box: coreos-stable
+  ram: 512
+  vcpu: 1
+  priv_ip: 192.168.1.106

consul-docker-registrator/user-data

@@ -0,0 +1,24 @@
+#cloud-config
+
+coreos:
+  units:
+    - name: etcd.service
+      command: stop
+      enable: false
+    - name: fleet.service
+      command: stop
+      enable: false
+    - name: docker-tcp.socket
+      command: start
+      enable: true
+      content: |
+        [Unit]
+        Description=Docker Socket for the API
+
+        [Socket]
+        ListenStream=2375
+        BindIPv6Only=both
+        Service=docker.service
+
+        [Install]
+        WantedBy=sockets.target