Finish Docker Swarm HA environment

Create ERB template for Consul configuration. Update Vagrantfile to
use ERB template. Update servers.yml to include role information.
Update Vagrantfile to perform provisioning based on role information.
Update consul.sh provisioning script to start Consul. Update
Vagrantfile to make Docker listen on network socket (and restart
Docker daemon).

docker-swarm-ha/README.md

@@ -1,17 +1,68 @@
-# Using Vagrant with OpenStack
+# Using Docker Swarm with Multiple Swarm Managers
 
-These files were created to allow users to use Vagrant ([http://www.vagrantup.com](http://www.vagrantup.com)) to automate the provisioning and de-provisioning of OpenStack instances. This configuration was tested using Vagrant 1.7.4 and version 0.7.0 of [the vagrant-openstack-provider plugin](https://github.com/ggiamarchi/vagrant-openstack-provider).
+These files were created to allow users to use Vagrant ([http://www.vagrantup.com](http://www.vagrantup.com)) to build an environment for working with Consul, Docker, Docker Swarm, and multiple Swarm manager instances. This configuration was tested using Vagrant 1.8.1, VMware Fusion 8.1.0, and version 4.0.5 of the Vagrant VMware plugin.
 
 ## Contents
 
+* **config.json.erb**: This is an ERB template used by Vagrant (and leveraging the information provided in `servers.yml`) to create VM-specific Consul configuration files. When you run `vagrant up` or `vagrant status`, this template will be used to create three VM-specific configuration files, each named "hostname.config.json", where "hostname" is the VM name specified in `servers.yml`. No edits to this file are needed.
+
+* **consul.conf**: This is an Ubuntu Upstart script for Consul. No changes or edits to this file are needed.
+
+* **consul.sh**: This Bash shell script is used as a provisioning tool by Vagrant when setting up the Consul VMs. No changes to this file are needed.
+
 * **README.md**: The file you're currently reading.
 
-* **Vagrantfile**: This file is used by Vagrant to spin up the OpenStack instance. Edit this file to change any of the properties of the OpenStack instance you want created by Vagrant.
+* **servers.yml**: This is a YAML file containing the configuration data used by Vagrant when creating and provisioning VMs. This Vagrant environment expects six (6) values in this file for each VM: name, Vagrant box, desired RAM, desired vCPUs, private IP address, and role (currently set to either "consul" or "docker"). At a minimum, you'll need to edit this file to specify the correct Vagrant box. Any other changes to this file are optional. Generally, the only other change that might be desired is to change the private IP addresses given to the VMs.
+
+* **Vagrantfile**: This file is used by Vagrant to spin up the virtual machines for this environment. No changes need to be made to this document, as all the configuration data is found in other files (like `servers.yml`). However, if you are using a virtualization solution _other_ than VMware Fusion, you might need to make changes to this file.
 
 ## Instructions
 
-Note that you'll need a working OpenStack installation in order to use these files.
+These instructions assume you've already installed VMware Fusion, Vagrant, and the Vagrant VMware plugin. Please refer to the documentation for those products for more information on installation or configuration. Note that Internet access is required when using `vagrant up` to create this environment.
 
-1. Edit `Vagrantfile` to ...
+1. Use `vagrant box add` to install an Ubuntu 14.04 x86_64 Vagrant box. I have a base box you can use for this purpose; to use my Ubuntu 14.04 x64 base box, add the box with `vagrant box add slowe/ubuntu-trusty-x64`.
+
+2. Place the files from the `docker-swarm-ha` directory of this GitHub repository into a directory on your local system. You can clone the entire "learning-tools" repository (using `git clone`) or just download the specific files from the the `docker-swarm-ha` folder.
+
+3. If necessary, edit `servers.yml` to specify the name of the Vagrant box downloaded in step 1. You may also make changes to the private IP addresses or the RAM/CPU settings, although no changes are required.
+
+4. Run `vagrant up` to instantiate the learning environment. This will spin up six (6) VMs based on the Ubuntu 14.04 base box you downloaded in step 1 and specified in `servers.yml` in step 3. Vagrant will also appropriately configure each VM and start the necessary services. Depending on the speed of your system and your Internet connection, this may take a few minutes.
+
+5. Use `vagrant ssh docker-01` to connect to the first Docker host and join the Docker Engine to a Swarm cluster:
+
+        docker run -d swarm join --addr=192.168.100.104 consul://192.168.100.101:8500/swarm
+
+    If you changed the private IP addresses in `servers.yml`, be sure to supply the appropriate IP addresses from that file in the command above.
+
+6. Repeat step #5 for the second and third Docker VMs, substituting the correct IP address each time. This means that the `--addr=` parameter changes, but the `consul://...` URL does _not_ change.
+
+7. Log back into the first Docker VM to start the first Docker Swarm manager:
+
+        docker run -d -p 3375:2375 swarm manage --replication \
+        --advertise 192.168.100.104:3375 consul://192.168.100.101:8500/swarm
+
+    You can use a port other than 3375 for the `-p` parameter, just be sure to match the specified port in the `--advertise` flag.
+
+8. Repeat step #7 on the second and third Docker VMs, changing the IP address specified for the `--advertise` parameter appropriately for each VM. Note that the URL for Consul does _not_ change.
+
+At this point you should have a working Docker Swarm cluster, backed by Consul, with one primary manager and two replica managers. To verify operation, run this command from one of the three Docker VMs:
+
+    docker -H tcp://192.168.100.104:3375 info
+
+This command should return information indicating that there are three Docker Engines in the Swarm cluster.
 
 Enjoy!
+
+## Troubleshooting
+
+* Verify that the Consul cluster is operating properly by querying Consul's HTTP API using `curl`. The following command should return a JSON-formatted list of the Consul cluster nodes:
+
+        curl -X GET http://192.168.100.101:8500/v1/catalog/nodes
+
+* After running the `docker run ... swarm join` commands, verify that the nodes are registering in Consul by examing the logs from the container. The output from the following command should include text that indicates the container is registering with the discovery service:
+
+        docker logs <docker container ID>
+
+## More Information
+
+Refer to the Docker documentation for a [manual Swarm install](https://docs.docker.com/swarm/install-manual/) and [working with multiple managers](https://docs.docker.com/swarm/multi-manager-setup/).

docker-swarm-ha/Vagrantfile

@@ -8,13 +8,36 @@ ENV['VAGRANT_VMWARE_CLONE_DIRECTORY'] = '~/.vagrant'
 ENV['VAGRANT_DEFAULT_PROVIDER'] = 'vmware_fusion'
 #ENV['VAGRANT_DEFAULT_PROVIDER'] = 'vmware_appcatalyst'
 
-# Require 'yaml' module
+# Require 'yaml', 'fileutils', and 'erb' modules
 require 'yaml'
+require 'fileutils'
+require 'erb'
 
 # Read YAML file with VM details (box, CPU, RAM, IP addresses)
 # Be sure to edit servers.yml to provide correct IP addresses
 servers = YAML.load_file('servers.yml')
 
+# Use template to create server-specific configuration files
+template = File.join(File.dirname(__FILE__), 'config.json.erb')
+content = ERB.new File.new(template).read
+
+# Populate cluster list
+cluster_list = []
+servers.each do |server|
+  if server['role'] == 'consul'
+    cluster_list << "\"#{server['priv_ip']}\""
+  end # if server['role']
+end # servers.each
+
+# Write out server-specific configuration files
+servers.each do |server|
+  if server['role'] == 'consul'
+    ip = server['priv_ip']
+    target = File.join(File.dirname(__FILE__), "#{server['name']}.config.json")
+    File.open(target, 'w') { |f| f.write(content.result(binding)) }
+  end # if server['role']
+end # servers.each
+
 # Create and configure the VMs
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
 
@@ -38,12 +61,21 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
       # Specify default synced folder; requires VMware Tools
       srv.vm.synced_folder '.', '/vagrant', disabled: true
 
-      # Copy in the Consul configuration files
-      srv.vm.provision 'file', source: 'server.json', destination: '/home/vagrant/config.json'
-      srv.vm.provision 'file', source: 'consul.conf', destination: '/home/vagrant/consul.conf'
+      # Configure VM based on "role" value from servers.yml
+      if server['role'] == 'consul'
+        # Copy Consul configuration files
+        srv.vm.provision 'file', source: "#{server['name']}.config.json", destination: '/home/vagrant/config.json'
+        srv.vm.provision 'file', source: 'consul.conf', destination: '/home/vagrant/consul.conf'
 
-      # Provision consul to the VMs
-      srv.vm.provision 'shell', path: 'consul.sh'
+        # Provision consul to the VMs
+        srv.vm.provision 'shell', path: 'consul.sh'
+      end # if server['role']
+
+      if server['role'] == 'docker'
+        srv.vm.provision 'docker', images: ['swarm']
+        srv.vm.provision 'shell', inline: 'echo DOCKER_OPTS=\"-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375\" >> /etc/default/docker', privileged: true
+        srv.vm.provision 'shell', inline: 'service docker restart', privileged: true
+      end
 
       # Configure VMs with RAM and CPUs per settings in servers.yml
       srv.vm.provider :vmware_fusion do |vmw|

docker-swarm-ha/config.json.erb

@@ -0,0 +1,11 @@
+{
+	"bootstrap_expect": 3,
+	"client_addr": "0.0.0.0",
+	"advertise_addr": "<%= server['priv_ip'] %>",
+	"server": true,
+	"datacenter": "dc1",
+	"data_dir": "/var/consul",
+	"log_level": "INFO",
+	"enable_syslog": true,
+	"start_join": [<%= cluster_list.join(',') %>]
+}

docker-swarm-ha/consul.sh

@@ -51,3 +51,6 @@ sudo mv /home/vagrant/consul.conf /etc/init/consul.conf
 sudo chown root:root /etc/init/consul.conf
 sudo mv /home/vagrant/config.json /etc/consul.d/server/config.json
 sudo chown root:consul /etc/consul.d/server/config.json
+
+# Start Consul
+sudo service consul start

docker-swarm-ha/server.json

@@ -1,9 +0,0 @@
-{
-	"bootstrap": false,
-	"server": true,
-	"datacenter": "dc1",
-	"data_dir": "/tmp",
-	"log_level": "INFO",
-	"enable_syslog": true,
-	"start_join": ["192.168.100.101", "192.168.100.102", "192.168.100.103"]
-}

docker-swarm-ha/servers.yml

@@ -4,13 +4,34 @@
   ram: "512"
   vcpu: "1"
   priv_ip: 192.168.100.101
+  role: "consul"
 - name: "consul-02"
   box: "slowe/ubuntu-trusty-x64"
   ram: "512"
   vcpu: "1"
   priv_ip: 192.168.100.102
+  role: "consul"
 - name: "consul-03"
   box: "slowe/ubuntu-trusty-x64"
   ram: "512"
   vcpu: "1"
   priv_ip: 192.168.100.103
+  role: "consul"
+- name: "docker-01"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  priv_ip: 192.168.100.104
+  role: "docker"
+- name: "docker-02"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  priv_ip: 192.168.100.105
+  role: "docker"
+- name: "docker-03"
+  box: "slowe/ubuntu-trusty-x64"
+  ram: "512"
+  vcpu: "1"
+  priv_ip: 192.168.100.106
+  role: "docker"