Add example of building EKS from scratch

Add Go example to show building EKS cluster from scratch with Pulumi.

Update README.md to reflect new Pulumi example.

Signed-off-by: Scott Lowe <scott.lowe@scottlowe.org>
This commit is contained in:
Scott Lowe 2023-11-02 13:53:08 -06:00 committed by Scott S. Lowe
parent 0ea8607133
commit fd04bd4592
7 changed files with 555 additions and 1 deletions

View file

@ -6,7 +6,9 @@ This folder contains environments, resources, and sample code for working with [
**aws-k8s-infra**: This folder contains example code to instantiate AWS infrastructure for use by Kubernetes clusters bootstrapped using `kubeadm`.
**default-aws-infra**: This folder contains example code to use the "default" infrastructure in your AWS account (the default VPC, subnets, Internet gateway, etc.).
**default-aws-infra**: This folder contains example Go code to use the "default" infrastructure in your AWS account (the default VPC, subnets, Internet gateway, etc.).
**eks-from-scratch**: This folder contains example Go code to stand up an AWS Elastic Kubernetes Service (EKS) cluster without using any component resources(such as AWSX or the EKS component).
**sandbox**: This folder contains Vagrant, Ansible, and Packer resources to create a sandbox for learning Pulumi.

View file

@ -0,0 +1,3 @@
name: eks-from-scratch
runtime: go
description: A Pulumi Go program to stand up an EKS cluster

View file

@ -0,0 +1,43 @@
# Building an AWS EKS Cluster from Scratch
This set of files provides an example on how to stand up an AWS Elastic Kubernetes Service (EKS) cluster "from scratch" with Pulumi. In this context, "from scratch" means not using any component resources such as those provided by the AWSX or EKS components.
This example uses [Go][link-1].
## Contents
* `go.mod`: This file contains dependencies used by this Go program.
* `go.sum`: This file contains checksums for each of the direct and indirect dependencies. The checksum is used to validate that none of them has been modified.
* `iam.go`: This Go file contains Pulumi code to create the necessary IAM elements for an EKS cluster. It is called by `main.go`.
* `main.go`: This Go file contains all the necessary Pulumi code to launch an EC2 instance on your default AWS infrastructure.
* `Pulumi.yaml`: This is the Pulumi project file.
* `README.md`: This file you're currently reading.
* `vpc.go`: This Go file contains Pulumi code to create a VPC and all necessary resources (subnets, gateways, etc.) for proper operation with an EKS cluster. It is called by `main.go`.
## Instructions
These instructions assume you've already installed and configured Pulumi and all necessary dependencies (Go, for this example). Please refer to the Pulumi documentation for more details on installation or configuration.
1. Copy the contents of this directory down to a directory on your system, or clone the entire repository and then change into the directory where this section of the cloned repository resides.
1. Run `pulumi stack init` to create a new stack.
1. Run `pulumi up` to instantiate the resources.
Enjoy! When you're finished, run `pulumi destroy` to tear down all the provisioned resources.
## Additional Notes
* The `vpc.go` file adds Kubernetes-related tags to resources to enable proper AWS integration. However, if you change the name of the cluster to something other than "testcluster" (this name is found in `main.go`), then be sure to adjust the tags appropriately (the cluster name is embedded in the tag names).
## License
This content is licensed under the MIT License.
[link-1]: https://go.dev

View file

@ -0,0 +1,87 @@
module eks-from-scratch
go 1.21
toolchain go1.21.0
require (
github.com/pulumi/pulumi-aws/sdk/v6 v6.6.0
github.com/pulumi/pulumi/sdk/v3 v3.86.0
)
require (
github.com/Microsoft/go-winio v0.5.2 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20221026131551-cf6655e29de4 // indirect
github.com/acomagu/bufpipe v1.0.3 // indirect
github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
github.com/agext/levenshtein v1.2.1 // indirect
github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
github.com/atotto/clipboard v0.1.4 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/blang/semver v3.5.1+incompatible // indirect
github.com/charmbracelet/bubbles v0.16.1 // indirect
github.com/charmbracelet/bubbletea v0.24.2 // indirect
github.com/charmbracelet/lipgloss v0.7.1 // indirect
github.com/cheggaaa/pb v1.0.29 // indirect
github.com/cloudflare/circl v1.3.3 // indirect
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
github.com/djherbis/times v1.5.0 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/go-git/gcfg v1.5.0 // indirect
github.com/go-git/go-billy/v5 v5.4.0 // indirect
github.com/go-git/go-git/v5 v5.6.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/glog v1.1.0 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/hcl/v2 v2.16.1 // indirect
github.com/imdario/mergo v0.3.13 // indirect
github.com/inconshreveable/mousetrap v1.0.1 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/mattn/go-isatty v0.0.18 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.14 // indirect
github.com/mitchellh/go-ps v1.0.0 // indirect
github.com/mitchellh/go-wordwrap v0.0.0-20150314170334-ad45545899c7 // indirect
github.com/muesli/ansi v0.0.0-20211018074035-2e021307bc4b // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/reflow v0.3.0 // indirect
github.com/muesli/termenv v0.15.1 // indirect
github.com/opentracing/basictracer-go v1.1.0 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/pjbgf/sha1cd v0.3.0 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pkg/term v1.1.0 // indirect
github.com/rivo/uniseg v0.4.4 // indirect
github.com/rogpeppe/go-internal v1.9.0 // indirect
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
github.com/santhosh-tekuri/jsonschema/v5 v5.0.0 // indirect
github.com/sergi/go-diff v1.2.0 // indirect
github.com/skeema/knownhosts v1.1.0 // indirect
github.com/spf13/cobra v1.6.1 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/texttheater/golang-levenshtein v1.0.1 // indirect
github.com/tweekmonster/luser v0.0.0-20161003172636-3fa38070dbd7 // indirect
github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/zclconf/go-cty v1.12.1 // indirect
go.uber.org/atomic v1.9.0 // indirect
golang.org/x/crypto v0.3.1-0.20221117191849-2c476679df9a // indirect
golang.org/x/net v0.10.0 // indirect
golang.org/x/sync v0.1.0 // indirect
golang.org/x/sys v0.8.0 // indirect
golang.org/x/term v0.8.0 // indirect
golang.org/x/text v0.9.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230706204954-ccb25ca9f130 // indirect
google.golang.org/grpc v1.57.0 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/frand v1.4.2 // indirect
sourcegraph.com/sourcegraph/appdash v0.0.0-20211028080628-e2786a622600 // indirect
)

View file

@ -0,0 +1,115 @@
package main
import (
"fmt"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/iam"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
// Define variables needed outside the createIam() function
var eksClusterRoleArn pulumi.StringInput
var eksNodeRoleArn pulumi.StringInput
func createIam(ctx *pulumi.Context) (err error) {
// Get policy documents for cluster and node assume role statements
// First for the cluster
clusterAssumeRole, err := iam.GetPolicyDocument(ctx, &iam.GetPolicyDocumentArgs{
Statements: []iam.GetPolicyDocumentStatement{
{
Effect: pulumi.StringRef("Allow"),
Principals: []iam.GetPolicyDocumentStatementPrincipal{
{
Type: "Service",
Identifiers: []string{
"eks.amazonaws.com",
},
},
},
Actions: []string{
"sts:AssumeRole",
},
},
},
}, nil)
if err != nil {
return err
}
// Second for the nodes
nodeAssumeRole, err := iam.GetPolicyDocument(ctx, &iam.GetPolicyDocumentArgs{
Statements: []iam.GetPolicyDocumentStatement{
{
Effect: pulumi.StringRef("Allow"),
Principals: []iam.GetPolicyDocumentStatementPrincipal{
{
Type: "Service",
Identifiers: []string{
"ec2.amazonaws.com",
},
},
},
Actions: []string{
"sts:AssumeRole",
},
},
},
}, nil)
if err != nil {
return err
}
// Define the cluster IAM role
clusterIamRole, err := iam.NewRole(ctx, "cluster-iam-role", &iam.RoleArgs{
AssumeRolePolicy: pulumi.String(clusterAssumeRole.Json),
})
if err != nil {
return err
}
// Define the node IAM role
nodeIamRole, err := iam.NewRole(ctx, "node-iam-role", &iam.RoleArgs{
AssumeRolePolicy: pulumi.String(nodeAssumeRole.Json),
})
if err != nil {
return err
}
// Attach the cluster IAM role to necessary policies
clusterPolicies := []string{
"arn:aws:iam::aws:policy/AmazonEKSClusterPolicy",
"arn:aws:iam::aws:policy/AmazonEKSVPCResourceController",
"arn:aws:iam::aws:policy/AmazonEKSServicePolicy",
}
for i, policy := range clusterPolicies {
_, err := iam.NewRolePolicyAttachment(ctx, fmt.Sprintf("cluster-pa-%d", i), &iam.RolePolicyAttachmentArgs{
PolicyArn: pulumi.String(policy),
Role: clusterIamRole.Name,
})
if err != nil {
return err
}
}
// Attach the node IAM role to necessary policies
nodePolicies := []string{
"arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
"arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy",
}
for i, policy := range nodePolicies {
_, err := iam.NewRolePolicyAttachment(ctx, fmt.Sprintf("node-pa-%d", i), &iam.RolePolicyAttachmentArgs{
PolicyArn: pulumi.String(policy),
Role: nodeIamRole.Name,
})
if err != nil {
return err
}
}
// Make the ARNs of the cluster and node IAM roles visible outside this function
eksClusterRoleArn = clusterIamRole.Arn
eksNodeRoleArn = nodeIamRole.Arn
return nil
}

View file

@ -0,0 +1,133 @@
package main
import (
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/eks"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
// Build out the base infrastructure (see vpc.go)
buildInfrastructure(ctx)
// Create IAM role (see iam.go)
createIam(ctx)
// Create a Security Group that we can use to actually connect to our cluster
clusterSg, err := ec2.NewSecurityGroup(ctx, "cluster-sg", &ec2.SecurityGroupArgs{
VpcId: vpcId,
Egress: ec2.SecurityGroupEgressArray{
ec2.SecurityGroupEgressArgs{
Protocol: pulumi.String("-1"),
FromPort: pulumi.Int(0),
ToPort: pulumi.Int(0),
CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
},
},
Ingress: ec2.SecurityGroupIngressArray{
ec2.SecurityGroupIngressArgs{
Protocol: pulumi.String("tcp"),
FromPort: pulumi.Int(80),
ToPort: pulumi.Int(80),
CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
},
ec2.SecurityGroupIngressArgs{
Protocol: pulumi.String("tcp"),
FromPort: pulumi.Int(443),
ToPort: pulumi.Int(443),
CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
},
},
})
if err != nil {
return err
}
// Create an EKS cluster
testCluster, err := eks.NewCluster(ctx, "test-cluster", &eks.ClusterArgs{
Name: pulumi.String("testcluster"),
RoleArn: eksClusterRoleArn,
VpcConfig: &eks.ClusterVpcConfigArgs{
EndpointPrivateAccess: pulumi.Bool(false),
EndpointPublicAccess: pulumi.Bool(true),
SecurityGroupIds: pulumi.StringArray{clusterSg.ID()},
SubnetIds: privateSubnets,
},
})
if err != nil {
return err
}
// Create a node group for the EKS cluster
_, err = eks.NewNodeGroup(ctx, "node-group", &eks.NodeGroupArgs{
ClusterName: testCluster.Name,
NodeRoleArn: eksNodeRoleArn,
SubnetIds: privateSubnets,
ScalingConfig: &eks.NodeGroupScalingConfigArgs{
DesiredSize: pulumi.Int(3),
MaxSize: pulumi.Int(6),
MinSize: pulumi.Int(3),
},
UpdateConfig: &eks.NodeGroupUpdateConfigArgs{
MaxUnavailable: pulumi.Int(1),
},
})
if err != nil {
return err
}
// Install the AWS EBS CSI addon
_, err = eks.NewAddon(ctx, "aws-ebs-csi", &eks.AddonArgs{
ClusterName: testCluster.Name,
AddonName: pulumi.String("aws-ebs-csi-driver"),
AddonVersion: pulumi.String("v1.24.0-eksbuild.1"),
ResolveConflictsOnUpdate: pulumi.String("PRESERVE"),
})
if err != nil {
return err
}
// Generate a Kubeconfig to access the cluster and make it accessible
clusterKubeconfig := generateKubeconfig(testCluster.Endpoint, testCluster.CertificateAuthority.Data().Elem(), testCluster.Name)
ctx.Export("kubeconfig", clusterKubeconfig)
return nil
})
}
// Create the KubeConfig structure as per https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
func generateKubeconfig(clusterEndpoint pulumi.StringOutput, certData pulumi.StringOutput, clusterName pulumi.StringOutput) pulumi.StringOutput {
return pulumi.Sprintf(`{
"apiVersion": "v1",
"clusters": [{
"cluster": {
"server": "%s",
"certificate-authority-data": "%s"
},
"name": "kubernetes",
}],
"contexts": [{
"context": {
"cluster": "kubernetes",
"user": "aws",
},
"name": "aws",
}],
"current-context": "aws",
"kind": "Config",
"users": [{
"name": "aws",
"user": {
"exec": {
"apiVersion": "client.authentication.k8s.io/v1beta1",
"command": "aws-iam-authenticator",
"args": [
"token",
"-i",
"%s",
],
},
},
}],
}`, clusterEndpoint, certData, clusterName)
}

View file

@ -0,0 +1,171 @@
package main
import (
"fmt"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
// Define variables needed outside the buildInfrastructure() function
var vpcId pulumi.StringInput
var publicSubnets pulumi.StringArray
var privateSubnets pulumi.StringArray
// Builds base infrastructure when called
func buildInfrastructure(ctx *pulumi.Context) (err error) {
// Look up AZ information for configured region and gather details
desiredAzState := "available"
rawAzInfo, err := aws.GetAvailabilityZones(ctx, &aws.GetAvailabilityZonesArgs{
State: &desiredAzState,
})
if err != nil {
return err
}
numOfAzs := len(rawAzInfo.Names)
azNames := make([]string, pulumi.Int(numOfAzs))
for i := 0; i < numOfAzs; i++ {
azNames[i] = rawAzInfo.Names[i]
}
// Create a new VPC and make the ID accessible outside the function
vpc, err := ec2.NewVpc(ctx, "vpc", &ec2.VpcArgs{
CidrBlock: pulumi.String("10.0.0.0/16"),
EnableDnsHostnames: pulumi.Bool(true),
EnableDnsSupport: pulumi.Bool(true),
Tags: pulumi.StringMap{
"kubernetes.io/cluster/testcluster": pulumi.String("shared"),
},
})
if err != nil {
return err
}
vpcId = vpc.ID()
// Create an Internet gateway
inetGw, err := ec2.NewInternetGateway(ctx, "inet-gw", &ec2.InternetGatewayArgs{
VpcId: vpc.ID(),
Tags: pulumi.StringMap{
"kubernetes.io/cluster/testcluster": pulumi.String("shared"),
},
})
if err != nil {
return err
}
// Adopt the default route in the VPC
defRoute, err := ec2.NewDefaultRouteTable(ctx, "def-route", &ec2.DefaultRouteTableArgs{
DefaultRouteTableId: vpc.DefaultRouteTableId,
Tags: pulumi.StringMap{
"kubernetes.io/cluster/testcluster": pulumi.String("shared"),
},
})
if err != nil {
return err
}
// Associate gateway with default route
_, err = ec2.NewRoute(ctx, "inet-route", &ec2.RouteArgs{
RouteTableId: defRoute.ID(),
DestinationCidrBlock: pulumi.String("0.0.0.0/0"),
GatewayId: inetGw.ID(),
})
if err != nil {
return err
}
// Create public subnets
for i := 0; i < numOfAzs; i++ {
subnetAddr := i * 32
subnetCidrBlock := fmt.Sprintf("10.0.%d.0/22", subnetAddr)
subnet, err := ec2.NewSubnet(ctx, fmt.Sprintf("pub-subnet-%d", i), &ec2.SubnetArgs{
VpcId: vpc.ID(),
AvailabilityZone: pulumi.String(azNames[i]),
CidrBlock: pulumi.String(subnetCidrBlock),
MapPublicIpOnLaunch: pulumi.Bool(true),
Tags: pulumi.StringMap{
"kubernetes.io/cluster/testcluster": pulumi.String("shared"),
"kubernetes.io/role/elb": pulumi.String("1"),
},
})
if err != nil {
return err
}
publicSubnets = append(publicSubnets, subnet.ID())
}
// Allocate Elastic IP for NAT Gateway for private subnets
eip, err := ec2.NewEip(ctx, "eip", &ec2.EipArgs{
Domain: pulumi.String("vpc"),
Tags: pulumi.StringMap{
"kubernetes.io/cluster/testcluster": pulumi.String("shared"),
},
})
if err != nil {
return err
}
// Create a NAT Gateway for the private subnets
natGw, err := ec2.NewNatGateway(ctx, "nat-gw", &ec2.NatGatewayArgs{
AllocationId: eip.ID(),
SubnetId: publicSubnets[0],
Tags: pulumi.StringMap{
"kubernetes.io/cluster/testcluster": pulumi.String("shared"),
},
}, pulumi.DependsOn([]pulumi.Resource{eip}))
if err != nil {
return err
}
// Create a route for the NAT Gateway
natRoute, err := ec2.NewRouteTable(ctx, "nat-route", &ec2.RouteTableArgs{
VpcId: vpc.ID(),
Routes: ec2.RouteTableRouteArray{
&ec2.RouteTableRouteArgs{
CidrBlock: pulumi.String("0.0.0.0/0"),
NatGatewayId: natGw.ID(),
},
},
Tags: pulumi.StringMap{
"kubernetes.io/cluster/testcluster": pulumi.String("shared"),
},
})
if err != nil {
return err
}
// Create private subnets
for i := 0; i < numOfAzs; i++ {
subnetAddr := (i * 32) + 16
subnetCidrBlock := fmt.Sprintf("10.0.%d.0/22", subnetAddr)
subnet, err := ec2.NewSubnet(ctx, fmt.Sprintf("priv-subnet-%d", i), &ec2.SubnetArgs{
VpcId: vpc.ID(),
AvailabilityZone: pulumi.String(azNames[i]),
CidrBlock: pulumi.String(subnetCidrBlock),
MapPublicIpOnLaunch: pulumi.Bool(false),
Tags: pulumi.StringMap{
"kubernetes.io/cluster/testcluster": pulumi.String("shared"),
"kubernetes.io/role/internal-elb": pulumi.String("1"),
},
})
if err != nil {
return err
}
privateSubnets = append(privateSubnets, subnet.ID())
}
// Link private subnets to private route table
for i := 0; i < numOfAzs; i++ {
_, err := ec2.NewRouteTableAssociation(ctx, fmt.Sprintf("priv-rta-%d", i), &ec2.RouteTableAssociationArgs{
SubnetId: privateSubnets[i],
RouteTableId: natRoute.ID(),
})
if err != nil {
return err
}
}
// Return to the calling function
return nil
}