Merge pull request #10 from lowescott/docker-swarm-ha

Docker Swarm Multiple Managers

docker-swarm-ha/README.md

@@ -0,0 +1,68 @@
+# Using Docker Swarm with Multiple Swarm Managers
+
+These files were created to allow users to use Vagrant ([http://www.vagrantup.com](http://www.vagrantup.com)) to build an environment for working with Consul, Docker, Docker Swarm, and multiple Swarm manager instances. This configuration was tested using Vagrant 1.8.1, VMware Fusion 8.1.0, and version 4.0.5 of the Vagrant VMware plugin.
+
+## Contents
+
+* **config.json.erb**: This is an ERB template used by Vagrant (and leveraging the information provided in `servers.yml`) to create VM-specific Consul configuration files. When you run `vagrant up` or `vagrant status`, this template will be used to create three VM-specific configuration files, each named "hostname.config.json", where "hostname" is the VM name specified in `servers.yml`. No edits to this file are needed.
+
+* **consul.conf**: This is an Ubuntu Upstart script for Consul. No changes or edits to this file are needed.
+
+* **consul.sh**: This Bash shell script is used as a provisioning tool by Vagrant when setting up the Consul VMs. No changes to this file are needed.
+
+* **README.md**: The file you're currently reading.
+
+* **servers.yml**: This is a YAML file containing the configuration data used by Vagrant when creating and provisioning VMs. This Vagrant environment expects six (6) values in this file for each VM: name, Vagrant box, desired RAM, desired vCPUs, private IP address, and role (currently set to either "consul" or "docker"). At a minimum, you'll need to edit this file to specify the correct Vagrant box. Any other changes to this file are optional. Generally, the only other change that might be desired is to change the private IP addresses given to the VMs.
+
+* **Vagrantfile**: This file is used by Vagrant to spin up the virtual machines for this environment. No changes need to be made to this document, as all the configuration data is found in other files (like `servers.yml`). However, if you are using a virtualization solution _other_ than VMware Fusion, you might need to make changes to this file.
+
+## Instructions
+
+These instructions assume you've already installed VMware Fusion, Vagrant, and the Vagrant VMware plugin. Please refer to the documentation for those products for more information on installation or configuration. Note that Internet access is required when using `vagrant up` to create this environment.
+
+1. Use `vagrant box add` to install an Ubuntu 14.04 x86_64 Vagrant box. I have a base box you can use for this purpose; to use my Ubuntu 14.04 x64 base box, add the box with `vagrant box add slowe/ubuntu-trusty-x64`.
+
+2. Place the files from the `docker-swarm-ha` directory of this GitHub repository into a directory on your local system. You can clone the entire "learning-tools" repository (using `git clone`) or just download the specific files from the the `docker-swarm-ha` folder.
+
+3. If necessary, edit `servers.yml` to specify the name of the Vagrant box downloaded in step 1. You may also make changes to the private IP addresses or the RAM/CPU settings, although no changes are required.
+
+4. Run `vagrant up` to instantiate the learning environment. This will spin up six (6) VMs based on the Ubuntu 14.04 base box you downloaded in step 1 and specified in `servers.yml` in step 3. Vagrant will also appropriately configure each VM and start the necessary services. Depending on the speed of your system and your Internet connection, this may take a few minutes.
+
+5. Use `vagrant ssh docker-01` to connect to the first Docker host and join the Docker Engine to a Swarm cluster:
+
+        docker run -d swarm join --addr=192.168.100.104 consul://192.168.100.101:8500/swarm
+
+    If you changed the private IP addresses in `servers.yml`, be sure to supply the appropriate IP addresses from that file in the command above.
+
+6. Repeat step #5 for the second and third Docker VMs, substituting the correct IP address each time. This means that the `--addr=` parameter changes, but the `consul://...` URL does _not_ change.
+
+7. Log back into the first Docker VM to start the first Docker Swarm manager:
+
+        docker run -d -p 3375:2375 swarm manage --replication \
+        --advertise 192.168.100.104:3375 consul://192.168.100.101:8500/swarm
+
+    You can use a port other than 3375 for the `-p` parameter, just be sure to match the specified port in the `--advertise` flag.
+
+8. Repeat step #7 on the second and third Docker VMs, changing the IP address specified for the `--advertise` parameter appropriately for each VM. Note that the URL for Consul does _not_ change.
+
+At this point you should have a working Docker Swarm cluster, backed by Consul, with one primary manager and two replica managers. To verify operation, run this command from one of the three Docker VMs:
+
+    docker -H tcp://192.168.100.104:3375 info
+
+This command should return information indicating that there are three Docker Engines in the Swarm cluster.
+
+Enjoy!
+
+## Troubleshooting
+
+* Verify that the Consul cluster is operating properly by querying Consul's HTTP API using `curl`. The following command should return a JSON-formatted list of the Consul cluster nodes:
+
+        curl -X GET http://192.168.100.101:8500/v1/catalog/nodes
+
+* After running the `docker run ... swarm join` commands, verify that the nodes are registering in Consul by examing the logs from the container. The output from the following command should include text that indicates the container is registering with the discovery service:
+
+        docker logs <docker container ID>
+
+## More Information
+
+Refer to the Docker documentation for a [manual Swarm install](https://docs.docker.com/swarm/install-manual/) and [working with multiple managers](https://docs.docker.com/swarm/multi-manager-setup/).

docker-swarm-ha/Vagrantfile

@@ -0,0 +1,93 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+# Specify Vagrant version, Vagrant API version, and Vagrant clone location
+Vagrant.require_version '>= 1.6.0'
+VAGRANTFILE_API_VERSION = '2'
+ENV['VAGRANT_VMWARE_CLONE_DIRECTORY'] = '~/.vagrant'
+ENV['VAGRANT_DEFAULT_PROVIDER'] = 'vmware_fusion'
+#ENV['VAGRANT_DEFAULT_PROVIDER'] = 'vmware_appcatalyst'
+
+# Require 'yaml', 'fileutils', and 'erb' modules
+require 'yaml'
+require 'fileutils'
+require 'erb'
+
+# Read YAML file with VM details (box, CPU, RAM, IP addresses)
+# Be sure to edit servers.yml to provide correct IP addresses
+servers = YAML.load_file('servers.yml')
+
+# Use template to create server-specific configuration files
+template = File.join(File.dirname(__FILE__), 'config.json.erb')
+content = ERB.new File.new(template).read
+
+# Populate cluster list
+cluster_list = []
+servers.each do |server|
+  if server['role'] == 'consul'
+    cluster_list << "\"#{server['priv_ip']}\""
+  end # if server['role']
+end # servers.each
+
+# Write out server-specific configuration files
+servers.each do |server|
+  if server['role'] == 'consul'
+    ip = server['priv_ip']
+    target = File.join(File.dirname(__FILE__), "#{server['name']}.config.json")
+    File.open(target, 'w') { |f| f.write(content.result(binding)) }
+  end # if server['role']
+end # servers.each
+
+# Create and configure the VMs
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+
+  # Always use Vagrant's default insecure key
+  config.ssh.insert_key = false
+
+  # Iterate through entries in YAML file to create VMs
+  servers.each do |server|
+    config.vm.define server['name'] do |srv|
+
+      # Don't check for box updates
+      srv.vm.box_check_update = false
+
+      # Specify hostname and Vagrant box
+      srv.vm.hostname = server['name']
+      srv.vm.box = server['box']
+
+      # Assign an additional static private network
+      srv.vm.network 'private_network', ip: server['priv_ip']
+
+      # Specify default synced folder; requires VMware Tools
+      srv.vm.synced_folder '.', '/vagrant', disabled: true
+
+      # Configure VM based on "role" value from servers.yml
+      if server['role'] == 'consul'
+        # Copy Consul configuration files
+        srv.vm.provision 'file', source: "#{server['name']}.config.json", destination: '/home/vagrant/config.json'
+        srv.vm.provision 'file', source: 'consul.conf', destination: '/home/vagrant/consul.conf'
+
+        # Provision consul to the VMs
+        srv.vm.provision 'shell', path: 'consul.sh'
+      end # if server['role']
+
+      if server['role'] == 'docker'
+        srv.vm.provision 'docker', images: ['swarm']
+        srv.vm.provision 'shell', inline: 'echo DOCKER_OPTS=\"-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375\" >> /etc/default/docker', privileged: true
+        srv.vm.provision 'shell', inline: 'service docker restart', privileged: true
+      end
+
+      # Configure VMs with RAM and CPUs per settings in servers.yml
+      srv.vm.provider :vmware_fusion do |vmw|
+        vmw.vmx['memsize'] = server['ram']
+        vmw.vmx['numvcpus'] = server['vcpu']
+      end # srv.vm.provider vmware_fusion
+
+      # Configure VMs with RAM and CPUs per settings in servers.yml
+      srv.vm.provider :vmware_appcatalyst do |vmw|
+        vmw.vmx['memsize'] = server['ram']
+        vmw.vmx['numvcpus'] = server['vcpu']
+      end # srv.vm.provider vmware_appcatalyst
+    end # config.vm.define
+  end # servers.each
+end # Vagrant.configure

docker-swarm-ha/config.json.erb

@@ -0,0 +1,11 @@
+{
+	"bootstrap_expect": 3,
+	"client_addr": "0.0.0.0",
+	"advertise_addr": "<%= server['priv_ip'] %>",
+	"server": true,
+	"datacenter": "dc1",
+	"data_dir": "/var/consul",
+	"log_level": "INFO",
+	"enable_syslog": true,
+	"start_join": [<%= cluster_list.join(',') %>]
+}

docker-swarm-ha/consul.conf

@@ -0,0 +1,19 @@
+description "Consul server process"
+
+start on runlevel [2345]
+stop on runlevel [!2345]
+
+respawn
+
+script
+  if [ -f "/etc/service/consul" ]; then
+    . /etc/service/consul
+  fi
+
+export GOMAXPROCS=`nproc`
+
+exec /usr/local/bin/consul agent \
+  -config-dir="/etc/consul.d/server" \
+  ${CONSUL_FLAGS} \
+  >>/var/log/consul.log 2>&1
+end script

docker-swarm-ha/consul.sh

@@ -0,0 +1,56 @@
+#!/bin/bash
+
+# Update list of packages
+export DEBIAN_FRONTEND=noninteractive
+sudo apt-get update
+
+# Install packages as needed
+if [[ ! -e /usr/bin/curl ]]; then
+  apt-get -yqq install curl
+fi
+
+if [[ ! -e /usr/bin/unzip ]]; then
+  apt-get -yqq install unzip
+fi
+
+# Download and install Consul binary, if needed
+if [[ ! -e /usr/local/bin/consul ]];then
+
+  # Download Consul
+  curl -kLO https://dl.bintray.com/mitchellh/consul/0.5.2_linux_amd64.zip
+
+  # Decompress and remove Consul download
+  unzip 0.5.2_linux_amd64.zip
+  rm 0.5.2_linux_amd64.zip
+
+  # Move Consul binary to location in path
+  sudo mv consul /usr/local/bin/
+fi
+
+# Create consul user, if it doesn't already exist
+if [ -z "$(getent passwd consul)" ]; then
+  useradd -M -d /var/consul -r -s /usr/bin/nologin consul
+ else
+   echo "Consul user already created."
+ fi
+
+# Create and configure Consul working directory
+sudo mkdir -p /var/consul
+if [ -d /var/consul ]; then
+  sudo chown -R consul:consul /var/consul
+fi
+
+# Create and configure Consul configuration directories
+sudo mkdir -p /etc/consul.d/server
+if [ -d /etc/consul.d ]; then
+  sudo chown -R root:consul /etc/consul.d
+fi
+
+# Move files into the correct locations
+sudo mv /home/vagrant/consul.conf /etc/init/consul.conf
+sudo chown root:root /etc/init/consul.conf
+sudo mv /home/vagrant/config.json /etc/consul.d/server/config.json
+sudo chown root:consul /etc/consul.d/server/config.json
+
+# Start Consul
+sudo service consul start

docker-swarm-ha/servers.yml

@@ -0,0 +1,37 @@
+---
+- name: "consul-01"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  priv_ip: 192.168.100.101
+  role: "consul"
+- name: "consul-02"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  priv_ip: 192.168.100.102
+  role: "consul"
+- name: "consul-03"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  priv_ip: 192.168.100.103
+  role: "consul"
+- name: "docker-01"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  priv_ip: 192.168.100.104
+  role: "docker"
+- name: "docker-02"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  priv_ip: 192.168.100.105
+  role: "docker"
+- name: "docker-03"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  priv_ip: 192.168.100.106
+  role: "docker"