prehistoric-systems/scottslowe-learning-tools
1
0
Fork 0
mirror of https://codeberg.org/scottslowe/learning-tools.git synced 2026-03-11 09:04:37 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add CAPI-Velero test infrastructure files

Browse source 
Add Terraform configuration for CAPI-Velero infrastructure. Add `kubeadm` configuration files for bootstrapping clusters.

Signed-off-by: Scott Lowe <scott.lowe@scottlowe.org>
This commit is contained in:
Scott Lowe 2021-01-20 21:23:19 -07:00
parent 51e351fd0c
commit 3743773945
No known key found for this signature in database
GPG key ID: 949F43F6E6C11780
12 changed files with 653 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

26
kubernetes/capi-velero/README.md Normal file
View file

@ -0,0 +1,26 @@
# Infrastructure for CAPI-Velero Testing
This repository contains the files necessary to create an environment for testing the use of [Velero](https://velero.io) with [Cluster API](https://cluster-api.sigs.k8s.io). This includes a [Terraform](https://terraform.io) configuration for creating the required AWS infrastructure, and `kubeadm` configuration files for bootstrapping the resulting instances into a [Kubernetes](https://kubernetes.io) cluster.
## Prerequisites/Assumptions
* This repository and these instructions assume the presence of IAM roles and policies that enable the AWS cloud provider. Refer to [this blog post](https://blog.scottlowe.org/2019/08/14/setting-up-aws-integrated-kubernetes-115-cluster-kubeadm/) for more information.
## Instructions
1. Create a `terraform.tfvars` file using the included `terraform.tfvars.example` file as an example.
2. Review the default values in `variables.tf` and override them, as needed, with additional values in the `terraform.tfvars` file created in step 1.
3. Review the IAM instance profile specified in `instances.tf` and make sure that the names there match the names of IAM instance profiles in your AWS account that enable/support the AWS cloud provider.
4. Run `terraform plan` and review the output.
5. If the output of step 4 is acceptable, run `terraform apply` to create the infrastructure.
6. Using SSH, log into each of the instances created in step 4 and ensure that the local hostname matches the EC2 Private DNS entry. Running `sudo hostnamectl set-hostname $(curl -s http://169.254.169.254/latest/meta-data/local-hostname)` will make sure the hostname is set correctly. (Note that SSH access is via a bastion host, so some local SSH configuration may be necessary.)
7. After step 5 and step 6 are completes, use `terraform output` to customize the `kubeadm-cp-mgmt-a.yaml` file. Specifically, change the value of the "controlPlaneEndpoint" line to reflect the correct DNS name of the load balancer created for the "mgmt-a" cluster. Also change the value of the "name" field under "nodeRegistration" from `HOSTNAME` to the full hostname set in step 6.
8. Repeat step 7, but for the `kubeadm-cp-mgmt-b.yaml` file and using the correct DNS name of the load balancer created for the "mgmt-b" cluster.
9. Use `rsync` or `scp` to copy the modified `kubeadm-cp-mgmt-a.yaml` and `kubeadm-cp-mgmt-b.yaml` to the control plane instances for the "mgmt-a" and "mgmt-b" clusters, respectively. Name the files `kubeadm.yaml` on the destination systems.
10. On the control plane instances for the "mgmt-a" and "mgmt-b" clusters (you can use `terraform output` to get the information on these instances), run `kubeadm init --config kubeadm.yaml`.
11. Using the information displayed by the `kubeadm init` command on each of the control plane nodes, customize the `kubeadm-wkr-mgmt-a.yaml` and `kubeadm-wkr-mgmt-b.yaml` files. Specifically, you will need to supply the value of the bootstrap token, the SHA256 hash of the control plane certificate, the DNS name of the control plane endpoint, and the hostname of the system (which, in step 6, you set to same value as the EC2 Private DNS entry).
12. Copy the files modified in step 11 to the worker instances for the "mgmt-a" and "mgmt-b" clusters, respectively. Name the files `kubeadm.yaml` on the destination systems.
13. On each of the worker instances, run `kubeadm join --config kubeadm.yaml`.
14. When step 13 completes, install a CNI plugin. Refer to the CNI plugin's documentation for specifics.
Upon the completion of the above steps, you will have a functional Kubernetes cluster that is ready to be made into a Cluster API management cluster using `clusterctl init`. These management clusters can then be used for testing the use of Velero for backing up and restoring Cluster API objects.

192
kubernetes/capi-velero/infra.tf Normal file
View file

@ -0,0 +1,192 @@
# Get AZ information
data "aws_availability_zones" "azinfo" {
state = "available"
}
# Create a new VPC
resource "aws_vpc" "velero-test-vpc" {
cidr_block = var.base_cidr
enable_dns_hostnames = "true"
enable_dns_support = "true"
tags = {
"Name" = "velero-test-vpc"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Create public subnets in the new VPC
resource "aws_subnet" "pub-subnet" {
count = length(data.aws_availability_zones.azinfo.names)
vpc_id = aws_vpc.velero-test-vpc.id
cidr_block = cidrsubnet(var.base_cidr, 4, count.index)
# cidr_block = format("%s.%d.0/20", var.base_cidr, 32*count.index)
availability_zone = data.aws_availability_zones.azinfo.names[count.index]
map_public_ip_on_launch = "true"
tags = {
"Name" = format("pub-subnet-%d", 1 + count.index)
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Create private subnets in the new VPC
resource "aws_subnet" "priv-subnet" {
count = length(data.aws_availability_zones.azinfo.names)
vpc_id = aws_vpc.velero-test-vpc.id
cidr_block = cidrsubnet(var.base_cidr, 4, 15 - count.index)
# cidr_block = format("%s.%d.0/20", var.base_cidr, 16+(32*count.index))
availability_zone = data.aws_availability_zones.azinfo.names[count.index]
map_public_ip_on_launch = "false"
tags = {
"Name" = format("priv-subnet-%d", 1 + count.index)
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Create an Internet gateway
resource "aws_internet_gateway" "inetgw" {
vpc_id = aws_vpc.velero-test-vpc.id
tags = {
"Name" = "velero-inetgw"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Create a route table for the public subnets
resource "aws_route_table" "inetgw-rte-tbl" {
vpc_id = aws_vpc.velero-test-vpc.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.inetgw.id
}
tags = {
"Name" = "inetgw-rte-tbl"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Associate the public subnets with the route table
resource "aws_route_table_association" "pub-rta" {
count = length(aws_subnet.pub-subnet[*].id)
subnet_id = aws_subnet.pub-subnet[count.index].id
route_table_id = aws_route_table.inetgw-rte-tbl.id
}
# Allocate an Elastic IP for the NAT gateway
resource "aws_eip" "eipnat" {
vpc = true
depends_on = [aws_internet_gateway.inetgw]
tags = {
"Name" = "eipnat"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Create a NAT gateway
resource "aws_nat_gateway" "natgw" {
allocation_id = aws_eip.eipnat.id
subnet_id = aws_subnet.pub-subnet[0].id
depends_on = [aws_internet_gateway.inetgw]
tags = {
"Name" = "velero-natgw"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Create a route table for the private subnets
resource "aws_route_table" "natgw-rte-tbl" {
vpc_id = aws_vpc.velero-test-vpc.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.natgw.id
}
depends_on = [aws_nat_gateway.natgw]
tags = {
"Name" = "inetgw-rte-tbl"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Associate the private subnets with the route table
resource "aws_route_table_association" "priv-rta" {
count = length(aws_subnet.priv-subnet[*].id)
subnet_id = aws_subnet.priv-subnet[count.index].id
route_table_id = aws_route_table.natgw-rte-tbl.id
}
# Create an ELB for mgmt cluster A
resource "aws_elb" "mgmt-a-elb" {
name = "mgmt-a-elb"
subnets = aws_subnet.pub-subnet.*.id
cross_zone_load_balancing = true
security_groups = [aws_security_group.mgmt-a-elb-sg.id]
listener {
instance_port = 6443
instance_protocol = "tcp"
lb_port = 6443
lb_protocol = "tcp"
}
health_check {
healthy_threshold = 3
unhealthy_threshold = 3
timeout = 5
target = "SSL:6443"
interval = 10
}
tags = {
"Name" = "mgmt-a-elb"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
}
}
# Create an ELB for mgmt cluster B
resource "aws_elb" "mgmt-b-elb" {
name = "mgmt-b-elb"
subnets = aws_subnet.pub-subnet.*.id
cross_zone_load_balancing = true
security_groups = [aws_security_group.mgmt-b-elb-sg.id]
listener {
instance_port = 6443
instance_protocol = "tcp"
lb_port = 6443
lb_protocol = "tcp"
}
health_check {
healthy_threshold = 3
unhealthy_threshold = 3
timeout = 5
target = "SSL:6443"
interval = 10
}
tags = {
"Name" = "mgmt-b-elb"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}

119
kubernetes/capi-velero/instances.tf Normal file
View file

@ -0,0 +1,119 @@
# Get the ID of a CAPA AMI
data "aws_ami" "capa_ami" {
most_recent = true
owners = ["258751437250"]
filter {
name = "name"
values = ["capa-ami-ubuntu-18.04-1.18.2*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
filter {
name = "root-device-type"
values = ["ebs"]
}
filter {
name = "architecture"
values = ["x86_64"]
}
}
# Launch an instance to serve as an SSH bastion host
resource "aws_instance" "bastion" {
ami = data.aws_ami.capa_ami.id
instance_type = var.bastion_instance_type
key_name = var.keypair
vpc_security_group_ids = [aws_security_group.bastion-sg.id]
subnet_id = aws_subnet.pub-subnet[0].id
depends_on = [aws_internet_gateway.inetgw]
tags = {
Name = "bastion-velero"
tool = "terraform"
demo = "velero-capi-testing"
area = "compute"
}
}
# Launch an instance to serve as a control plane node for mgmt cluster A
resource "aws_instance" "cp-mgmt-a" {
ami = data.aws_ami.capa_ami.id
instance_type = var.cp_instance_type
key_name = var.keypair
vpc_security_group_ids = [aws_security_group.controlplane-sg.id]
subnet_id = aws_subnet.priv-subnet[0].id
iam_instance_profile = "control-plane.cluster-api-provider-aws.sigs.k8s.io"
depends_on = [aws_internet_gateway.inetgw]
tags = {
"Name" = "cp-mgmt-a-velero"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
}
}
# Attach mgmt cluster A control plane node to mgmt cluster A ELB
resource "aws_elb_attachment" "mgmt-a-elb-attach" {
elb = aws_elb.mgmt-a-elb.id
instance = aws_instance.cp-mgmt-a.id
depends_on = [aws_elb.mgmt-a-elb]
}
# Launch an instance to serve as a control plane node for mgmt cluster B
resource "aws_instance" "cp-mgmt-b" {
ami = data.aws_ami.capa_ami.id
instance_type = var.cp_instance_type
key_name = var.keypair
vpc_security_group_ids = [aws_security_group.controlplane-sg.id]
subnet_id = aws_subnet.priv-subnet[0].id
iam_instance_profile = "control-plane.cluster-api-provider-aws.sigs.k8s.io"
depends_on = [aws_internet_gateway.inetgw]
tags = {
"Name" = "cp-mgmt-b-velero"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Attach mgmt cluster B control plane node to mgmt cluster B ELB
resource "aws_elb_attachment" "mgmt-b-elb-attach" {
elb = aws_elb.mgmt-b-elb.id
instance = aws_instance.cp-mgmt-b.id
depends_on = [aws_elb.mgmt-b-elb]
}
# Launch an instance to serve as a worker node in mgmt cluster A
resource "aws_instance" "wkr-mgmt-a" {
ami = data.aws_ami.capa_ami.id
instance_type = var.cp_instance_type
key_name = var.keypair
vpc_security_group_ids = [aws_security_group.worker-sg.id]
subnet_id = aws_subnet.priv-subnet[0].id
iam_instance_profile = "nodes.cluster-api-provider-aws.sigs.k8s.io"
depends_on = [aws_internet_gateway.inetgw]
tags = {
"Name" = "wkr-mgmt-a-velero"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
}
}
# Launch an instance to serve as a worker node in mgmt cluster B
resource "aws_instance" "wkr-mgmt-b" {
ami = data.aws_ami.capa_ami.id
instance_type = var.cp_instance_type
key_name = var.keypair
vpc_security_group_ids = [aws_security_group.worker-sg.id]
subnet_id = aws_subnet.priv-subnet[0].id
iam_instance_profile = "nodes.cluster-api-provider-aws.sigs.k8s.io"
depends_on = [aws_internet_gateway.inetgw]
tags = {
"Name" = "wkr-mgmt-b-velero"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}

24
kubernetes/capi-velero/kubeadm-cp-mgmt-a.yaml Normal file
View file

@ -0,0 +1,24 @@
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
extraArgs:
cloud-provider: aws
clusterName: mgmt-a
controlPlaneEndpoint: "mgmt-a-elb-random.region.elb.amazonaws.com:6443"
controllerManager:
extraArgs:
cloud-provider: aws
configure-cloud-routes: "false"
kubernetesVersion: v1.18.2
networking:
dnsDomain: cluster.local
podSubnet: 192.168.0.0/16
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
name: HOSTNAME
kubeletExtraArgs:
cloud-provider: aws

24
kubernetes/capi-velero/kubeadm-cp-mgmt-b.yaml Normal file
View file

@ -0,0 +1,24 @@
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
extraArgs:
cloud-provider: aws
clusterName: mgmt-b
controlPlaneEndpoint: "mgmt-b-elb-random.region.elb.amazonaws.com:6443"
controllerManager:
extraArgs:
cloud-provider: aws
configure-cloud-routes: "false"
kubernetesVersion: v1.18.2
networking:
dnsDomain: cluster.local
podSubnet: 192.168.0.0/16
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
name: HOSTNAME
kubeletExtraArgs:
cloud-provider: aws

12
kubernetes/capi-velero/kubeadm-wkr-mgmt-a.yaml Normal file
View file

@ -0,0 +1,12 @@
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
bootstrapToken:
token: TOKEN
apiServerEndpoint: "mgmt-a-elb-random.region.elb.amazonaws.com:6443"
caCertHashes: ["HASH"]
nodeRegistration:
name: HOSTNAME
kubeletExtraArgs:
cloud-provider: aws

12
kubernetes/capi-velero/kubeadm-wkr-mgmt-b.yaml Normal file
View file

@ -0,0 +1,12 @@
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
bootstrapToken:
token: TOKEN
apiServerEndpoint: "mgmt-b-elb-random.region.elb.amazonaws.com:6443"
caCertHashes: ["HASH"]
nodeRegistration:
name: HOSTNAME
kubeletExtraArgs:
cloud-provider: aws

39
kubernetes/capi-velero/output.tf Normal file
View file

@ -0,0 +1,39 @@
output "vpc_id" {
value = aws_vpc.velero-test-vpc.id
}
output "pub_subnet_ids" {
value = [aws_subnet.pub-subnet[*].id]
}
output "priv_subnet_ids" {
value = [aws_subnet.priv-subnet[*].id]
}
output "bastion_pub_ip" {
value = aws_instance.bastion.public_ip
}
output "mgmt_a_cp_priv_ip" {
value = aws_instance.cp-mgmt-a.private_ip
}
output "wkr_a_cp_priv_ip" {
value = aws_instance.wkr-mgmt-a.private_ip
}
output "mgmt_a_elb_dns" {
value = aws_elb.mgmt-a-elb.dns_name
}
output "mgmt_b_cp_priv_ip" {
value = aws_instance.cp-mgmt-b.private_ip
}
output "wkr_b_cp_priv_ip" {
value = aws_instance.wkr-mgmt-b.private_ip
}
output "mgmt_b_elb_dns" {
value = aws_elb.mgmt-b-elb.dns_name
}

12
kubernetes/capi-velero/provider.tf Normal file
View file

@ -0,0 +1,12 @@
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
}
provider "aws" {
region = var.user_region
}

149
kubernetes/capi-velero/securitygroups.tf Normal file
View file

@ -0,0 +1,149 @@
# Create a security group for the SSH bastion host
resource "aws_security_group" "bastion-sg" {
vpc_id = aws_vpc.velero-test-vpc.id
name = "bastion-sg"
description = "Security group for SSH bastion hosts"
ingress {
from_port = "0"
to_port = "0"
protocol = "-1"
cidr_blocks = [aws_vpc.velero-test-vpc.cidr_block]
}
ingress {
from_port = "22"
to_port = "22"
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = "0"
to_port = "0"
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
"Name" = "bastion-sg"
"tool" = "terraform"
"demo" = "velero-capi-testing"
}
}
# Create a security group for the ELB for management cluster A
resource "aws_security_group" "mgmt-a-elb-sg" {
vpc_id = aws_vpc.velero-test-vpc.id
name = "mgmt-a-elb-sg"
description = "ELB security group for mgmt cluster A"
ingress {
from_port = "6443"
to_port = "6443"
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = "6443"
to_port = "6443"
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
"Name" = "mgmt-a-elb-sg"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
}
}
# Create a security group for the ELB for management cluster B
resource "aws_security_group" "mgmt-b-elb-sg" {
vpc_id = aws_vpc.velero-test-vpc.id
name = "mgmt-b-elb-sg"
description = "ELB security group for mgmt cluster B"
ingress {
from_port = "6443"
to_port = "6443"
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = "6443"
to_port = "6443"
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
"Name" = "mgmt-b-elb-sg"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Create a security group for control plane nodes
resource "aws_security_group" "controlplane-sg" {
vpc_id = aws_vpc.velero-test-vpc.id
name = "controlplane-sg"
description = "Security group for K8s control plane nodes"
ingress {
from_port = "0"
to_port = "0"
protocol = "-1"
cidr_blocks = [aws_vpc.velero-test-vpc.cidr_block]
}
ingress {
from_port = "6443"
to_port = "6443"
protocol = "tcp"
security_groups = [aws_security_group.mgmt-a-elb-sg.id, aws_security_group.mgmt-b-elb-sg.id]
}
ingress {
from_port = "22"
to_port = "22"
protocol = "tcp"
security_groups = [aws_security_group.bastion-sg.id]
}
egress {
from_port = "0"
to_port = "0"
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
"Name" = "controlplane-sg"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}
# Create a security group for K8s worker nodes
resource "aws_security_group" "worker-sg" {
vpc_id = aws_vpc.velero-test-vpc.id
name = "worker-sg"
description = "Security group for K8s worker nodes"
ingress {
from_port = "0"
to_port = "0"
protocol = "-1"
cidr_blocks = [aws_vpc.velero-test-vpc.cidr_block]
}
ingress {
from_port = "22"
to_port = "22"
protocol = "tcp"
security_groups = [aws_security_group.bastion-sg.id]
}
egress {
from_port = "0"
to_port = "0"
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
"Name" = "worker-sg"
"tool" = "terraform"
"demo" = "velero-capi-testing"
"kubernetes.io/cluster/mgmt-a" = "shared"
"kubernetes.io/cluster/mgmt-b" = "shared"
}
}

6
kubernetes/capi-velero/terraform.tfvars.example Normal file
View file

@ -0,0 +1,6 @@
keypair = "id_rsa_aws"
cp_instance_type = "t2.large"
wkr_instance_type = "t2.xlarge"
bastion_instance_type = "t2.small"
user_region = "us-east-2"
base_cidr = "10.54.0.0/16"

38
kubernetes/capi-velero/variables.tf Normal file
View file

@ -0,0 +1,38 @@
variable "keypair" {
type = string
description = "SSH keypair to use to connect to instances"
}
variable "cp_instance_type" {
type = string
description = "AWS type to use when creating manager instances"
default = "t2.medium"
}
variable "wkr_instance_type" {
type = string
description = "AWS type to use when creating worker instances"
default = "t2.large"
}
variable "bastion_instance_type" {
type = string
description = "AWS type to use when creating SSH bastion instances"
default = "t2.small"
}
variable "user_region" {
type = string
description = "AWS region to use for new resources"
}
variable "num_wkr_nodes" {
type = number
description = "Number of worker nodes to create"
default = 1
}
variable "base_cidr" {
type = string
description = "Base CIDR for the infrastructure"
}