Create environment for Consul using Ansible

Create new learning environment for Consul that uses Ansible for
provisioning instead of a shell script.

consul-ansible/README.md

@@ -0,0 +1,43 @@
+# Running a Consul Cluster in Vagrant
+
+These files were created to allow users to use Vagrant ([http://www.vagrantup.com](http://www.vagrantup.com)) quickly and relatively easily spin up a Consul ([http://www.consul.io](http://www.consul.io)) cluster. The configuration was tested using Vagrant 1.7.2, VMware Fusion 6.0.5, and the Vagrant VMware plugin.
+
+## Contents
+
+* **ansible.cfg**: This Ansible configuration file tells Ansible which inventory file to use (a file named `hosts` in the current directory), which SSH private key to use (the default Vagrant insecure key), and which remote user to use (the default user of `vagrant` that should be common to all Vagrant boxes). No edits to this file should be necessary.
+
+* **config.json.j2**: This Jinja2 template is automatically filled in with appropriate values by Ansible when the Ansible playbook is applied against the Vagrant VMs. No edits to this file should be necessary.
+
+* **consul.conf**: This Upstart script configures Consul to run as a background daemon (service) on the Ubuntu-based VMs created by Vagrant. This file is installed by Ansible when the Ansible playbook is applied against the Vagrant VMs. No edits to this file should be necessary.
+
+* **hosts**: This Ansible inventory file is generated automatically by Vagrant once you run `vagrant status` or any other command that requires Vagrant to parse the `Vagrantfile`. Since it is automatically generated, no edits directly to this file are needed (they would be overwritten anyway).
+
+* **machines.yml**: This YAML file contains a list of VM definitions. It is referenced by `Vagrantfile` when Vagrant instantiates the VMs. Generally, the only change needed to this file is to specify the correct Vagrant box you will be used (see "Instructions" below). If necessary, you may need to edit the IP addresses supplied in this file to avoid IP addressing conflicts with other networks.
+
+* **provision.yml**: This is the Ansible playbook that will configure the Vagrant VMs to become members of a Consul cluster. No edits to this file should be necessary.
+
+* **README.md**: This file you're currently reading.
+
+* **Vagrantfile**: This file is used by Vagrant to spin up the virtual machines. This file is fairly extensively commented to help explain what's happening. You should be able to use this file unchanged; all the VM configuration options are stored outside this file.
+
+## Instructions
+
+These instructions assume you've already installed VMware Fusion, Vagrant, and the Vagrant VMware plugin. Please refer to the documentation for those products for more information on installation or configuration.
+
+1. Use `vagrant box add` to install an Ubuntu 14.04 x64 box for the vmware_fusion provider. I have a base box you can use for this purpose; to use my Ubuntu 14.04 x64 base box, add the box with `vagrant box add slowe/ubuntu-trusty-x64`.
+
+2. Place the files from the `consul` directory of this GitHub repository into a directory on your local system. You can clone the entire "learning-tools" repository (using `git clone`) or just download the specific files from the the `consul` folder.
+
+3. If you are using a Vagrant box other than my Ubuntu 14.04 base box (referred to in step #1), edit the `machines.yml` file to specify the box in use. If necessary to avoid IP address conflicts with existing networks, you may also need to edit the IP addresses specified in this file. Generally, no other changes are needed, although (if you are comfortable with the settings) you can adjust the number of virtual CPUs and/or the amount of RAM assigned to each Vagrant VM in this file as well. Note that this environment _assumes_ the presence of an `eth1` in each Vagrant VM; therefore, do not remove the "ip_addr" value from `machines.yml`.
+
+4. Once you have edited `machines.yml`, use `vagrant up` to bring up the 3 systems that will serve as your Consul cluster.
+
+5. Once Vagrant has finished bringing up the VMs, run `ansible-playbook provision.yml`. Ansible will use the configuration file in the current directory to pull inventory from the file named `hosts` (it was automatically generated when you ran `vagrant up`). This will provision and configure Consul on each of the Vagrant VMs.
+
+    **NOTE**: Due to the way the Ansible provisioner in Vagrant works, it's currently not possible to provision the VMs with Ansible from within the `Vagrantfile`. As a result, step #5 (running `ansible-playbook` manually) is needed.
+
+At this point, you have a functional Consul cluster running under Vagrant. If you are using VMware Fusion, you should have IP connectivity to the VMs, and can use the OS X `consul` binary to connect to the cluster and test it. For example, this command would work to demonstrate that Consul is working (you would need to change the IP address provided after `-rpc-addr`):
+
+	consul members -rpc-addr=192.168.1.101:8400
+
+Enjoy!

consul-ansible/Vagrantfile

@@ -0,0 +1,79 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# Specify Vagrant version, Vagrant API version, and desired clone location
+Vagrant.require_version '>= 1.6.0'
+VAGRANTFILE_API_VERSION = '2'
+ENV['VAGRANT_DEFAULT_PROVIDER'] = 'vmware_fusion'
+ENV['VAGRANT_VMWARE_CLONE_DIRECTORY'] = '~/.vagrant'
+#ENV['VAGRANT_DEFAULT_PROVIDER'] = 'virtualbox'
+#ENV['VAGRANT_DEFAULT_PROVIDER'] = 'vmware_appcatalyst'
+
+# Require 'yaml' module
+require 'yaml'
+
+# Read YAML file with VM details (box, CPU, RAM, IP addresses)
+machines = YAML.load_file(File.join(File.dirname(__FILE__), 'machines.yml'))
+
+# Create custom Ansible inventory file (can't use built-in inventory)
+f = File.open('hosts','w')
+machines.each do |machine|
+  line = machine['name'] + "     ansible_ssh_host=" + machine['ip_addr']
+  f.puts line
+end # machines.each
+f.close
+
+# Create and configure the VMs
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+
+  # Always use Vagrant's default insecure key
+  config.ssh.insert_key = false
+
+  # Iterate through entries in YAML file to create VMs
+  machines.each do |machine|
+    config.vm.define machine['name'] do |srv|
+
+      # Don't check for box updates
+      srv.vm.box_check_update = false
+      srv.vm.hostname = machine['name']
+      srv.vm.box = machine['box']
+
+      # Configure default synced folder (disable by default)
+      if machine['sync_disabled'] != nil
+        srv.vm.synced_folder '.', '/vagrant', disabled: machine['sync_disabled']
+      else
+        srv.vm.synced_folder '.', '/vagrant', disabled: true
+      end #if machine['sync_disabled']
+
+      # Assign additional private network
+      if machine['ip_addr'] != nil
+        srv.vm.network 'private_network', ip: machine['ip_addr']
+      end # if machine['ip_addr']
+
+      # Configure CPU & RAM per settings in machines.yml (Fusion)
+      srv.vm.provider :vmware_fusion do |vmw|
+        vmw.vmx['memsize'] = machine['ram']
+        vmw.vmx['numvcpus'] = machine['vcpu']
+        if machine['nested'] == true
+          vmw.vmx['vhv.enable'] = 'TRUE'
+        end #if machine['nested']
+      end # srv.vm.provider vmware_fusion
+
+      # Configure CPU & RAM per settings in machines.yml (AppCatalyst)
+      srv.vm.provider :vmware_appcatalyst do |vmw|
+        vmw.vmx['memsize'] = machine['ram']
+        vmw.vmx['numvcpus'] = machine['vcpu']
+        #vmw.vmx['guestos'] = 'other3xlinux-64'
+        if machine['nested'] == true
+          vmw.vmx['vhv.enable'] = 'TRUE'
+        end #if machine['nested']
+      end # srv.vm.provider vmware_appcatalyst
+
+      # Configure CPU & RAM per settings in machines.yml (VirtualBox)
+      srv.vm.provider :virtualbox do |vb|
+        vb.memory = machine['ram']
+        vb.cpus = machine['vcpu']
+      end # srv.vm.provider virtualbox
+    end # config.vm.define
+  end # machines.each
+end # Vagrant.configure

consul-ansible/ansible.cfg

@@ -0,0 +1,5 @@
+[defaults]
+inventory = ./hosts
+private_key_file = ~/.vagrant.d/insecure_private_key
+remote_user = vagrant
+host_key_checking = False

consul-ansible/config.json.j2

@@ -0,0 +1,19 @@
+{% set lbracket = "[" %}
+{% set rbracket = "]" %}
+{% set quote = '"' %}
+
+{
+	"bootstrap_expect": 3,
+	"server": true,
+	"datacenter": "dc1",
+	"data_dir": "/var/consul",
+	"log_level": "INFO",
+	"enable_syslog": false,
+	"retry_join": [ {% for host in groups['all'] %}{% if not loop.last %}
+"{{ hostvars[host]['ansible_eth1']['ipv4']['address'] }}", 
+      {% else %}
+"{{ hostvars[host]['ansible_eth1']['ipv4']['address'] }}"
+      {% endif %}{% endfor %} ],
+	"client_addr": "0.0.0.0",
+	"advertise_addr": "{{ hostvars[inventory_hostname]['ansible_eth1']['ipv4']['address'] }}"
+}

consul-ansible/consul.conf

@@ -0,0 +1,19 @@
+description "Consul server process"
+
+start on runlevel [2345]
+stop on runlevel [!2345]
+
+respawn
+
+script
+  if [ -f "/etc/service/consul" ]; then
+    . /etc/service/consul
+  fi
+
+export GOMAXPROCS=`nproc`
+
+exec /usr/local/bin/consul agent \
+  -config-dir="/etc/consul.d/server" \
+  ${CONSUL_FLAGS} \
+  >>/var/log/consul.log 2>&1
+end script

consul-ansible/hosts

@@ -0,0 +1,3 @@
+consul-01     ansible_ssh_host=192.168.100.101
+consul-02     ansible_ssh_host=192.168.100.102
+consul-03     ansible_ssh_host=192.168.100.103

consul-ansible/machines.yml

@@ -0,0 +1,16 @@
+---
+- name: "consul-01"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  ip_addr: 192.168.100.101
+- name: "consul-02"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  ip_addr: 192.168.100.102
+- name: "consul-03"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  ip_addr: 192.168.100.103

consul-ansible/provision.yml

@@ -0,0 +1,73 @@
+---
+- hosts: "all"
+  sudo: "yes"
+  remote_user: "vagrant"
+
+  tasks:
+  - name: Remove i386 architecture to eliminate apt-get errors
+    command: "/usr/bin/dpkg --remove-architecture i386"
+
+  - name: Install unzip package
+    apt:
+      pkg: "unzip"
+      state: "latest"
+      update_cache: "yes"
+
+  - name: Create consul group
+    group:
+      name: "consul"
+      state: "present"
+
+  - name: Create user for Consul
+    user:
+      name: "consul"
+      group: "consul"
+      comment: "Consul daemon user"
+      shell: "/usr/sbin/nologin"
+      state: "present"
+      home: "/var/consul"
+      system: "yes"
+      createhome: "no"
+
+  - name: Create Consul data and configuration directories
+    file:
+      path: "{{ item }}"
+      state: "directory"
+      owner: "consul"
+      group: "consul"
+    with_items:
+      - "/var/consul"
+      - "/etc/consul.d/server"
+
+  - name: Download Consul package
+    get_url:
+      url: "https://releases.hashicorp.com/consul/0.6.0/consul_0.6.0_linux_amd64.zip"
+      dest: "/tmp/consul_0.6.0_linux_amd64.zip"
+
+  - name: Unzip downloaded Consul file
+    unarchive:
+      copy: "no"
+      src: "/tmp/consul_0.6.0_linux_amd64.zip"
+      dest: "/usr/local/bin/"
+      creates: "/usr/local/bin/consul"
+
+  - name: Install Upstart script for Consul
+    copy:
+      src: "consul.conf"
+      dest: "/etc/init/consul.conf"
+      owner: "root"
+      group: "root"
+      mode: "0644"
+
+  - name: Install Consul configuration file
+    template:
+      src: "config.json.j2"
+      dest: "/etc/consul.d/server/config.json"
+      owner: "consul"
+      group: "consul"
+      mode: "0644"
+
+  - name: Start Consul service
+    service:
+      name: "consul"
+      state: "started"