Update Docker Swarm-etcd environment

Update the etcd-backed Docker Swarm environment with new methods/techniques learned in building the Vagrant environment. This involves changes to the external YAML file (renamed to "machines.yml") and the Vagrantfile. Only minor changes were made to the ERB template and the provisioning script (to eliminate a dependency on shared folders).

Signed-off-by: Scott S. Lowe <scott.lowe@scottlowe.org>
This commit is contained in:
Scott S. Lowe 2016-10-20 16:18:13 -06:00
parent 8afb4b147b
commit 4f0eb76317
6 changed files with 145 additions and 112 deletions

View file

@ -1,18 +1,18 @@
# Running an etcd-Backed Docker Swarm Cluster in Vagrant
These files were created to allow users to use Vagrant ([http://www.vagrantup.com](http://www.vagrantup.com)) quickly and relatively easily spin up a Docker Swarm ([http://docs.docker.com/swarm/](http://docs.docker.com/swarm/)) cluster backed by etcd ([https://github.com/coreos/etcd/](https://github.com/coreos/etcd/)). The configuration was tested using Vagrant 1.7.2, VMware Fusion 6.0.5, and the Vagrant VMware plugin.
These files were created to allow users to use Vagrant ([http://www.vagrantup.com](http://www.vagrantup.com)) quickly and relatively easily spin up a Docker Swarm ([http://docs.docker.com/swarm/](http://docs.docker.com/swarm/)) cluster backed by etcd ([https://github.com/coreos/etcd/](https://github.com/coreos/etcd/)). The configuration was tested using Vagrant 1.8.6, VMware Fusion 8.1.0 with the Vagrant VMware plugin, and VirtualBox 5.1.
## Contents
* **etcd.conf**: This is an Upstart script (written for Ubuntu) to start etcd. No modifications to this file should be necessary. This file is installed by the provisioning script called in `Vagrantfile`.
* **etcd.defaults.erb**: A template used by Vagrant to create machine-specific service defaults files for each node in the etcd cluster. The appropriate node-specific file is installed as `etcd` in the `/etc/default` directory on each node by the provisioning script called in `Vagrantfile`. These files are generated for each VM whose "etcd" value in `servers.yml` is set to "active".
* **etcd.defaults.erb**: A template used by Vagrant to create machine-specific service defaults files for each node in the etcd cluster. The appropriate node-specific file is installed as `etcd` in the `/etc/default` directory on each node by the provisioning script called in `Vagrantfile`. These files are generated for each VM whose "etcd" value in `machines.yml` is set to "true".
* **provision.sh**: This provisioning script is called by the shell provisioner in `Vagrantfile`. It downloads the etcd 2.0.9 release from GitHub, expands it, creates necessary directories, and places files in the appropriate locations. Finally it starts etcd (or restarts it if already running).
* **provision.sh**: This provisioning script is called by the shell provisioner in `Vagrantfile`. It downloads the etcd 2.3.7 release from GitHub, expands it, creates necessary directories, and places files in the appropriate locations. Finally it starts etcd (or restarts it if already running).
* **README.md**: This file you're currently reading.
* **servers.yml**: This YAML file contains a list of VM definitions and associated configuration data. It is referenced by `Vagrantfile` when Vagrant instantiates the VMs. You _might_ need to edit this file to change the IP addresses (the default IP addresses may not work in all environments).
* **machines.yml**: This YAML file contains a list of VM definitions and associated configuration data. It is referenced by `Vagrantfile` when Vagrant instantiates the VMs. You _might_ need to edit this file to change the IP addresses (the default IP addresses may not work in all environments).
* **user_data**: This file is used by the CoreOS cloud-init process to customize the CoreOS VMs upon instantiation. This file disables etcd and fleet, and configures Docker to listen on TCP port 2375.
@ -20,52 +20,52 @@ These files were created to allow users to use Vagrant ([http://www.vagrantup.co
## Instructions
These instructions assume you've already installed VMware Fusion, Vagrant, and the Vagrant VMware plugin. Please refer to the documentation for those products for more information on installation or configuration.
These instructions assume you've already installed your virtualization provider (VMware Fusion/Workstation or VirtualBox), Vagrant, and any applicable Vagrant plugins (like the Vagrant VMware plugin). Please refer to the documentation for those products for more information on installation or configuration.
1. Use `vagrant box add` to install an Ubuntu 14.04 x64 box for the vmware_fusion provider. I have a base box you can use for this purpose; to use my Ubuntu 14.04 x64 base box, add the box with `vagrant box add slowe/ubuntu-trusty-x64`.
1. Use `vagrant box add` to install an Ubuntu 14.04 x64 box for your virtualization provider. If you are using VMware Fusion/Workstation, I have a base box you can use for this purpose; to use my Ubuntu 14.04 x64 base box, add the box with `vagrant box add slowe/ubuntu-trusty-x64`.
2. Use `vagrant box add` to install a CoreOS base box. The `Vagrantfile` assumes you are using the Stable release of CoreOS (the box named `coreos-stable`). If the box name is different, you'll need to edit both `servers.yml` and the `Vagrantfile` to specify the correct box name. You'll need to be sure to use a version of CoreOS that comes with Docker 1.4.0 or later, as Swarm requires Docker => 1.4.0. Note that CoreOS Stable 557.2.0 comes with Docker 1.4.1.
2. Use `vagrant box add` to install a CoreOS base box. The `Vagrantfile` assumes you are using the Stable release of CoreOS (the box named `coreos-stable`). If the box name is different, you'll need to edit both `machines.yml` and the `Vagrantfile` to specify the correct box name. You'll need to be sure to use a version of CoreOS that comes with Docker 1.4.0 or later, as Swarm requires Docker => 1.4.0. Note that CoreOS Stable 557.2.0 comes with Docker 1.4.1.
3. Place the files from the `docker-swarm-etcd` directory of this GitHub repository into a directory on your local system. You can clone the entire "learning-tools" repository (using `git clone`) or just download the specific files from the the `docker-swarm-etcd` folder.
4. Edit the `servers.yml` file to make sure the box names specified in this file match the names of the boxes you added to your system in steps 1 and 2. _Generally speaking, no other changes to this file should be necessary._ The `Vagrantfile` expects six values for each VM: `name` (the user-friendly name of the VM, which will also be used as the hostname for the guest OS inside the VM); `box` (the name of an Ubuntu 14.04 base box); `ram` (the amount of memory to be assigned to the VM); `vcpu` (the number of vCPUs that should be assigned to the VM); `priv_ip` (an IP address to be statically assigned to the VM and is used for Consul cluster communications); and `etcd` (set to "active" for a node to be included in the etcd cluster configuration; any other value will exclude it from the etcd cluster configuration).
4. Edit the `machines.yml` file to make sure the box names specified in this file match the names of the boxes you added to your system in steps 1 and 2. _Generally speaking, no other changes to this file should be necessary._ The `Vagrantfile` expects six values for each VM: `name` (the user-friendly name of the VM, which will also be used as the hostname for the guest OS inside the VM); `box` (two subvalues which provide the name of the Vagrant box to use for the VMware and VirtualBox providers); `ram` (the amount of memory to be assigned to the VM); `vcpu` (the number of vCPUs that should be assigned to the VM); `ip_addr` (an IP address to be statically assigned to the VM and is used for Consul cluster communications); and `etcd` (set to "true" for a node to be included in the etcd cluster configuration; any other value will exclude it from the etcd cluster configuration).
5. Edit `Vagrantfile` to make sure the box names referenced in this file (on lines 55 and 64) match the names of the boxes you added to your system in steps 1 and 2.
5. Edit `Vagrantfile` to make sure the box names referenced in this file (on line 90) match the names of the boxes you added to your system in steps 1 and 2.
6. Once you have edited `servers.yml` (and `Vagrantfile`, if necessary), use `vagrant up` to bring up the 6 systems. Three VMs will run the etcd cluster; the other 3 VMs will be running CoreOS and will make up the Docker Swarm cluster.
6. Once you have edited `machines.yml` (and `Vagrantfile`, if necessary), use `vagrant up` to bring up the 6 systems. Three VMs will run the etcd cluster; the other 3 VMs will be running CoreOS and will make up the Docker Swarm cluster.
7. Once Vagrant has finished bringing up the VMs, simply use `vagrant ssh etcd-01` (where `etcd-01` is the value assigned to the first VM from `servers.yml`) to connect to the VM. No password should be required; it should use the default (insecure) SSH key. Once you are logged into the first VM, verify the operation of etcd with this command:
7. Once Vagrant has finished bringing up the VMs, simply use `vagrant ssh etcd-01` (where `etcd-01` is the value assigned to the first VM from `machines.yml`) to connect to the VM. No password should be required; it should use the default (insecure) SSH key. Once you are logged into the first VM, verify the operation of etcd with this command:
etcdctl member list
This command should return a list of three members, which are the first three VMs (based on an Ubuntu 14.04 box) specified in `servers.yml`. If the command does not return a list of three members, then you'll need to troubleshoot etcd.
This command should return a list of three members, which are the first three VMs (based on an Ubuntu 14.04 box) specified in `machines.yml`. If the command does not return a list of three members, then you'll need to troubleshoot etcd.
8. Use `vagrant ssh coreos-01` to log into the first CoreOS system (`coreos-01` is the default name supplied in `servers.yml`; if you've changed the default name, modify your command appropriately). Add it as the first node to a new Docker Swarm cluster using this command:
8. Use `vagrant ssh coreos-01` to log into the first CoreOS system (`coreos-01` is the default name supplied in `machines.yml`; if you've changed the default name, modify your command appropriately). Add it as the first node to a new Docker Swarm cluster using this command:
docker run -d swarm join --addr=192.168.101.111:2375 etcd://192.168.101.101:2379/swarm
docker run -d swarm join --addr=192.168.100.111:2375 etcd://192.168.100.101:2379/swarm
Make sure the IP address referenced by the `--addr=` parameter matches the IP address assigned to the CoreOS VM (which is pulled from the "priv_ip" key in `servers.yml`). Further, make sure the IP address in the "etcd://" URL matches the IP address assigned to one of the Ubuntu-based etcd cluster nodes.
Make sure the IP address referenced by the `--addr=` parameter matches the IP address assigned to the CoreOS VM (which is pulled from the "ip_addr" key in `machines.yml`). Further, make sure the IP address in the "etcd://" URL matches the IP address assigned to one of the Ubuntu-based etcd cluster nodes.
9. Log out of the first CoreOS system and use `vagrant ssh` to log into the second and third CoreOS systems, repeating step 8 on each system. Be sure to change values for the `--addr=` parameter on each system. The default values from `servers.yml` use the addresses 192.168.101.111 (coreos-01), 192.168.101.112 (coreos-02), and 192.168.101.113 (coreos-03).
9. Log out of the first CoreOS system and use `vagrant ssh` to log into the second and third CoreOS systems, repeating step 8 on each system. Be sure to change values for the `--addr=` parameter on each system. The default values from `machines.yml` use the addresses 192.168.100.111 (coreos-01), 192.168.100.112 (coreos-02), and 192.168.100.113 (coreos-03).
9. On the first CoreOS system (the one named `coreos-01` by default), run a Docker container that will serve as the manager of the Swarm cluster. Launch the Swarm manager with this command:
docker run -d -p 8333:2375 swarm manage etcd://192.168.101.101:2379/swarm
docker run -d -p 8333:2375 swarm manage etcd://192.168.100.101:2379/swarm
The IP address specified in the "etcd://" URL should correspond to the IP address assigned to one of the Ubuntu-based etcd cluster nodes. Make note of the port exposed by the `-p` command and the IP address of the node on which you're running the Swarm manager. _This will be the endpoint against which you will run Docker commands against the cluster._
10. Verify the operation of the Swarm cluster by running this command (from any system that has connectivity to the CoreOS system running the Swarm manager container launched in the previous step):
docker -H tcp://192.168.101.111:8333 info
docker -H tcp://192.168.100.111:8333 info
Docker should return information indicating that there are 4 containers running across 3 nodes, and then provide more information about each node and the containers running on that node.
11. Launch an Nginx container somewhere on the cluster with this command:
docker -H tcp://192.168.101.111:8333 run -d --name www -p 80:80 nginx
docker -H tcp://192.168.100.111:8333 run -d --name www -p 80:80 nginx
If everything is working as expected, Docker will launch an Nginx container on one of the CoreOS nodes in the Swarm cluster. Re-running the command from step 10 can help track it down, as well as using this command:
docker -H tcp://192.168.101.111:8333 ps
docker -H tcp://192.168.100.111:8333 ps
Enjoy!

View file

@ -1,12 +1,9 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Specify Vagrant version, Vagrant API version, and Vagrant clone location
Vagrant.require_version ">= 1.6.0"
VAGRANTFILE_API_VERSION = "2"
ENV['VAGRANT_VMWARE_CLONE_DIRECTORY'] = '~/.vagrant'
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'vmware_fusion'
#ENV['VAGRANT_DEFAULT_PROVIDER'] = 'virtualbox'
# Specify minimum Vagrant version and Vagrant API version
Vagrant.require_version '>= 1.6.0'
VAGRANTFILE_API_VERSION = '2'
# Require 'yaml', 'fileutils', and 'erb' modules
require 'yaml'
@ -15,26 +12,27 @@ require 'erb'
# Look for user-data file to configure/customize CoreOS boxes
# No changes should need to be made to this file
USER_DATA = File.join(File.dirname(__FILE__), "user-data")
USER_DATA = File.join(File.dirname(__FILE__), 'user-data')
# Read YAML file with VM details (box, CPU, RAM, IP addresses)
# Be sure to edit servers.yml to provide correct IP addresses
servers = YAML.load_file(File.join(File.dirname(__FILE__), 'servers.yml'))
# Be sure to edit machines.yml to provide correct IP addresses
machines = YAML.load_file(File.join(File.dirname(__FILE__), 'machines.yml'))
# Use config from YAML file to write out templates for etcd overrides
template = File.join(File.dirname(__FILE__), 'etcd.defaults.erb')
content = ERB.new File.new(template).read
# Build etcd override files from ERB template
etcd_initial_cluster = []
servers.each do |servers|
if servers["etcd"] == "active"
etcd_initial_cluster << "#{servers['name']}=http://#{servers['priv_ip']}:2380"
machines.each do |machine|
if machine['etcd'] == true
etcd_initial_cluster << "#{machine['name']}=http://#{machine['ip_addr']}:2380"
end
end
servers.each do |servers|
if servers["etcd"] == "active"
ip = servers['priv_ip']
target = File.join(File.dirname(__FILE__), "#{servers['name']}.defaults")
machines.each do |machine|
if machine['etcd'] == true
ip = machine['ip_addr']
target = File.join(File.dirname(__FILE__), "#{machine['name']}.defaults")
File.open(target, 'w') { |f| f.write(content.result(binding)) }
end
end
@ -46,45 +44,68 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.ssh.insert_key = false
# Iterate through entries in YAML file to create VMs
servers.each do |servers|
config.vm.define servers["name"] do |srv|
machines.each do |machine|
# Configure the VMs per details in machines.yml
config.vm.define machine['name'] do |srv|
# Don't check for box updates
srv.vm.box_check_update = false
srv.vm.hostname = servers["name"]
srv.vm.box = servers["box"]
# Assign an additional static private network
srv.vm.network "private_network", ip: servers["priv_ip"]
# Specify the hostname of the VM
srv.vm.hostname = machine['name']
# Specify the Vagrant box to use (use VMware box by default)
srv.vm.box = machine['box']['vmw']
# Configure default synced folder (disable by default)
if machine['sync_disabled'] != nil
srv.vm.synced_folder '.', '/vagrant', disabled: machine['sync_disabled']
else
srv.vm.synced_folder '.', '/vagrant', disabled: true
end #if machine['sync_disabled']
# Assign additional private network
if machine['ip_addr'] != nil
srv.vm.network 'private_network', ip: machine['ip_addr']
end # if machine['ip_addr']
# Configure CPU & RAM per settings in machines.yml (Fusion)
srv.vm.provider 'vmware_fusion' do |vmw|
vmw.vmx['memsize'] = machine['ram']
vmw.vmx['numvcpus'] = machine['vcpu']
if machine['nested'] == true
vmw.vmx['vhv.enable'] = 'TRUE'
end #if machine['nested']
end # srv.vm.provider 'vmware_fusion'
# Configure CPU & RAM per settings in machines.yml (VirtualBox)
srv.vm.provider 'virtualbox' do |vb, override|
vb.memory = machine['ram']
vb.cpus = machine['vcpu']
override.vm.box = machine['box']['vb']
end # srv.vm.provider 'virtualbox'
# Configure VMs based on CoreOS box
if srv.vm.box == "coreos-stable"
# Disable default synced folder for CoreOS VMs
srv.vm.synced_folder ".", "/vagrant", disabled: true
if srv.vm.box == 'coreos-stable'
# Copy user_data file into CoreOS VM
srv.vm.provision "file", source: "#{USER_DATA}", destination: "/tmp/vagrantfile-user-data"
srv.vm.provision 'file', source: "#{USER_DATA}", destination: '/tmp/vagrantfile-user-data'
# Move user_data to correct location to be processed by cloud-init
srv.vm.provision "shell", inline: "mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/", privileged: true
srv.vm.provision 'shell', inline: 'mv /tmp/vagrantfile-user-data /var/lib/coreos-vagrant/', privileged: true
end # if srv.vm.box
# Configure VMs based on Ubuntu box
if srv.vm.box == "slowe/ubuntu-trusty-x64"
# Enable default synced folder (needed for provisioning)
srv.vm.synced_folder ".", "/vagrant"
# Run final provisioning script
# Note that default shared folder is needed for script to work
srv.vm.provision "shell", path: "provision.sh", privileged: true
# Configure VMs marked with etcd value of true
if machine['etcd'] == true
# Copy files into the VM
srv.vm.provision 'file', source: 'etcd.conf', destination: '/home/vagrant/etcd.conf'
srv.vm.provision 'file', source: "#{machine['name']}.defaults", destination: "/home/vagrant/#{machine['name']}.defaults"
# Run etcd provisioning script
srv.vm.provision 'shell', path: 'provision.sh', privileged: true
end # if srv.vm.box
# Configure VMs with RAM and CPUs per settings in servers.yml (Fusion)
srv.vm.provider :vmware_fusion do |vmw|
vmw.vmx["memsize"] = servers["ram"]
vmw.vmx["numvcpus"] = servers["vcpu"]
end # srv.vm.provider vmware_fusion
# Configure VMs with RAM and CPUs per settings in servers.yml (VirtualBox)
srv.vm.provider :virtualbox do |vb|
vb.memory = servers["ram"]
vb.cpus = servers["vcpu"]
end # srv.vm.provider virtualbox
end # config.vm.define
end # servers.each
end # machines.each
end # Vagrant.configure

View file

@ -5,9 +5,9 @@
export ETCD_INITIAL_CLUSTER="<%= etcd_initial_cluster.join(',') %>"
export ETCD_INITIAL_CLUSTER_STATE="new"
export ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
export ETCD_INITIAL_ADVERTISE_PEER_URLS="http://<%= servers['priv_ip'] %>:2380"
export ETCD_INITIAL_ADVERTISE_PEER_URLS="http://<%= machine['ip_addr'] %>:2380"
export ETCD_DATA_DIR="/var/etcd"
export ETCD_LISTEN_PEER_URLS="http://<%= servers['priv_ip'] %>:2380"
export ETCD_LISTEN_CLIENT_URLS="http://<%= servers['priv_ip'] %>:2379,http://127.0.0.1:2379"
export ETCD_ADVERTISE_CLIENT_URLS="http://<%= servers['priv_ip'] %>:2379"
export ETCD_NAME="<%= servers['name'] %>"
export ETCD_LISTEN_PEER_URLS="http://<%= machine['ip_addr'] %>:2380"
export ETCD_LISTEN_CLIENT_URLS="http://<%= machine['ip_addr'] %>:2379,http://127.0.0.1:2379"
export ETCD_ADVERTISE_CLIENT_URLS="http://<%= machine['ip_addr'] %>:2379"
export ETCD_NAME="<%= machine['name'] %>"

View file

@ -0,0 +1,49 @@
---
- name: "etcd-01"
box:
vmw: "slowe/ubuntu-trusty-x64"
vb: "ubuntu/trusty64"
ram: "512"
vcpu: "1"
ip_addr: "192.168.100.101"
etcd: true
- name: "etcd-02"
box:
vmw: "slowe/ubuntu-trusty-x64"
vb: "ubuntu/trusty64"
ram: "512"
vcpu: "1"
ip_addr: "192.168.100.102"
etcd: true
- name: "etcd-03"
box:
vmw: "slowe/ubuntu-trusty-x64"
vb: "ubuntu/trusty64"
ram: "512"
vcpu: "1"
ip_addr: "192.168.100.103"
etcd: true
- name: "coreos-01"
box:
vmw: "coreos-stable"
vb: "coreos-stable"
ram: "512"
vcpu: "1"
ip_addr: "192.168.100.111"
etcd: false
- name: "coreos-02"
box:
vmw: "coreos-stable"
vb: "coreos-stable"
ram: "512"
vcpu: "1"
ip_addr: "192.168.100.112"
etcd: false
- name: "coreos-03"
box:
vmw: "coreos-stable"
vb: "coreos-stable"
ram: "512"
vcpu: "1"
ip_addr: "192.168.100.113"
etcd: false

View file

@ -1,7 +1,7 @@
#!/bin/bash
# Define a couple variables for easier future modifications
VERS="v2.0.9"
VERS="v2.3.7"
BASENAME="etcd-$VERS-linux-amd64"
FNAME="$BASENAME.tar.gz"
URL="https://github.com/coreos/etcd/releases/download/$VERS/$FNAME"
@ -38,8 +38,8 @@ if [[ ! -d /var/etcd ]]; then
fi
# Copy files into the correct locations; requires shared folders
sudo cp /vagrant/etcd.conf /etc/init/etcd.conf
sudo cp /vagrant/$HOSTNAME.defaults /etc/default/etcd
sudo cp /home/vagrant/etcd.conf /etc/init/etcd.conf
sudo cp /home/vagrant/$HOSTNAME.defaults /etc/default/etcd
# Restart if already running; otherwise, start etcd.
sudo initctl status etcd && initctl restart etcd || initctl start etcd

View file

@ -1,37 +0,0 @@
---
- name: etcd-01
box: slowe/ubuntu-trusty-x64
ram: 512
vcpu: 1
priv_ip: 192.168.101.101
etcd: active
- name: etcd-02
box: slowe/ubuntu-trusty-x64
ram: 512
vcpu: 1
priv_ip: 192.168.101.102
etcd: active
- name: etcd-03
box: slowe/ubuntu-trusty-x64
ram: 512
vcpu: 1
priv_ip: 192.168.101.103
etcd: active
- name: coreos-01
box: coreos-stable
ram: 512
vcpu: 1
priv_ip: 192.168.101.111
etcd: inactive
- name: coreos-02
box: coreos-stable
ram: 512
vcpu: 1
priv_ip: 192.168.101.112
etcd: inactive
- name: coreos-03
box: coreos-stable
ram: 512
vcpu: 1
priv_ip: 192.168.101.113
etcd: inactive