2 posts tagged with "CloudFormation"

In today's fast-paced tech world, automating infrastructure setup is key to maximizing efficiency and reducing human error. One of the most reliable tools for this is AWS CloudFormation, which allows users to define their cloud resources and manage them as code. While AWS provides a Console for managing CloudFormation, the AWS Command Line Interface (CLI) is a powerful alternative that offers speed, control, and flexibility. In this blog, we'll walk you through setting up CloudFormation using AWS CLI, covering essential components like VPCs, KMS keys, and IAM roles.

1. Introduction to AWS CloudFormation​

Before diving into technical details, it's important to understand what AWS CloudFormation is and why it's so beneficial.

What is AWS CloudFormation?​

AWS CloudFormation is an Infrastructure-as-Code (IaC) service provided by AWS that allows you to model, provision, and manage AWS and third-party resources. You define your resources using template files (JSON or YAML) and deploy them via AWS CloudFormation, which takes care of the provisioning and configuration.

CloudFormation manages the entire lifecycle of your resources, from creation to deletion, allowing for automation and consistent environments.

Benefits of Using CloudFormation​

1. Automation: CloudFormation automates the entire infrastructure setup, from VPC creation to IAM role configuration, reducing manual work and errors.

2. Version Control: Treat your infrastructure like code. With CloudFormation, you can manage your infrastructure in repositories like Git, making it easy to version, track, and rollback changes.

3. Consistency: CloudFormation ensures that the same template can be used to create identical environments, such as development, staging, and production.

4. Cost Efficiency: With CloudFormation, resources can be automatically deleted when no longer needed, preventing unnecessary costs from unused resources.

2. Why Use AWS CLI Over the Console?​

AWS CLI vs Console: Which One is Better for You?​

The AWS Management Console offers an intuitive, visual interface for managing AWS resources, but it's not always the most efficient way to manage infrastructure, especially when it grows complex. Here's how AWS CLI compares:

Advantages of Using AWS CLI​

1. Automation: CLI commands can be scripted for automation, allowing you to run tasks without manually navigating through the Console.
2. Faster Setup: CLI allows you to automate stack creation, updates, and deletion, significantly speeding up the setup process.
3. Better Error Handling: You can incrementally update stacks and fix errors on the go with AWS CLI, making it easier to debug and manage resources.

3. Prerequisites​

Before we start building with CloudFormation, let’s go over the prerequisites.

Setting Up AWS CLI​

AWS CLI is a powerful tool that allows you to manage AWS services from the command line. To get started:

1. Install AWS CLI:

   • If you haven't already installed AWS CLI, you can do so by following the instructions from the official AWS documentation.

2. Verify Installation: After installation, verify that the AWS CLI is installed by typing the following command in your terminal:

   aws --version
   

   If successfully installed, the version information will be displayed.

Configuring AWS Profiles​

Before using AWS CLI to interact with your AWS account, you'll need to configure a profile:

aws configure

You'll be prompted to provide:

• AWS Access Key ID
• AWS Secret Access Key
• Default region name (e.g., us-west-2)
• Default output format (choose JSON)

This configuration will allow the CLI to authenticate and interact with your AWS account.

4. Step-by-Step Guide to AWS CloudFormation with AWS CLI​

Now that your CLI is set up, let us get into how to deploy AWS CloudFormation stacks using it.

Setting Up Your First CloudFormation Stack​

We will start with a simple example of how to create a CloudFormation stack. Suppose you want to create a Virtual Private Cloud (VPC).

1. Create a YAML Template: Save the following template in a file named vpc.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      Tags:
        - Key: Name
          Value: MyVPC

2. Deploy the Stack: To create the VPC, run the following command:

aws cloudformation create-stack --stack-name my-vpc-stack --template-body file://vpc.yaml --capabilities CAPABILITY_NAMED_IAM

This command will instruct CloudFormation to spin up a VPC using the specified template.

3. Check the Stack Status: To verify the status of your stack creation, use:

aws cloudformation describe-stacks --stack-name my-vpc-stack

Deploying a Virtual Private Cloud (VPC)​

A VPC is essential for defining your network infrastructure in AWS. Here’s how you can add more resources to your VPC, such as an Internet Gateway:

Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16
      Tags:
        - Key: Name
          Value: MyVPC
  InternetGateway:
    Type: AWS::EC2::InternetGateway
  VPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref MyVPC
      InternetGatewayId: !Ref InternetGateway

Deploy this using the same create-stack command.

Setting Up Security with KMS (Key Management Service)​

Next, we will add encryption keys for securing data:

1. KMS Template:

Resources:
  MyKMSKey:
    Type: AWS::KMS::Key
    Properties:
      Description: Key for encrypting data
      Enabled: true

2. Deploy KMS:

aws cloudformation create-stack --stack-name my-kms-stack --template-body file://kms.yaml --capabilities CAPABILITY_NAMED_IAM

Managing Access with IAM Roles​

IAM Roles allow secure communication between AWS services. Here’s an example of how to create an IAM role:

Resources:
  MyIAMRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      Path: /

Use the same create-stack command to deploy this.

5. Best Practices for AWS CloudFormation​

Use Nested Stacks​

Avoid large, monolithic stacks. Break them down into smaller, nested stacks for better manageability.

Resources:
  ParentStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/path/to/nested-stack.yaml

Parameterization​

Use parameters to make your stacks reusable across different environments.

Parameters:
  InstanceType:
    Type: String
    Default: t2.micro
    Description: EC2 Instance Type

Exporting and Referencing Outputs​

Export important resource values for use in other stacks:

Outputs:
  VPCId:
    Value: !Ref MyVPC
    Export:
      Name: VPCId

Incremental Stack Updates​

Always update your stacks incrementally to avoid failures.

aws cloudformation update-stack --stack-name my-stack --template-body file://updated-template.yaml

6. Advanced CloudFormation Features​

Handling Dependencies and Stack Failures​

Use the DependsOn attribute to specify dependencies between resources to avoid issues with resource creation order.

Custom Resource Creation​

For advanced use cases, you can create custom resources by using Lambda functions or CLI.

7. Conclusion and Next Steps​

By using AWS CloudFormation with AWS CLI, you can automate your infrastructure, reduce errors, and scale your environment effortlessly. Continue learning by experimenting with more complex templates, incorporating advanced features like stack sets, and automating further with scripts.

Code shown in the video can be accessed from https://github.com/arinatechnologies/cloudformation

Introduction​

This document outlines the steps taken to deploy and configure GitLab Runners, including the installation of Terraform, ensuring that the application team can focus solely on writing pipelines.

Architecture​

The following diagram displays the solution architecture.

AWS CloudFormation is used to create the infrastructure hosting the GitLab Runner. The main steps are as follows:

1. The user runs a deploy script to deploy the CloudFormation template. The template is parameterized, and the parameters are defined in a properties file. The properties file specifies the infrastructure configuration and the environment in which to deploy the template.
2. The deploy script calls CloudFormation CreateStack API to create a GitLab Runner stack in the specified environment.
3. During stack creation, an EC2 autoscaling group is created with the desired number of EC2 instances. Each instance is launched via a launch template created with values from the properties file. An IAM role is created and attached to the EC2 instance, containing permissions required for the GitLab Runner to execute pipeline jobs. A lifecycle hook is attached to the autoscaling group on instance termination events, ensuring graceful instance termination.
4. During instance launch, GitLab Runner will be configured and installed. Terraform, Git, and other software will also be installed as needed.
5. The user may repeat the same steps to deploy GitLab Runner into another environment.

Infrastructure Setup with CloudFormation​

Customizing the CloudFormation Template​

The initial step in deploying GitLab Runners involved setting up the infrastructure using AWS CloudFormation. The standard CloudFormation template was customized to fit the unique requirements of the environment.

CloudFormation Template Location: GitLab Runner Template

CloudFormation Template Location: GitLab Runner Scaling Group / Cluster Template

For any automation requirement or issues, please reach out to us Contact Us

Parameters used:

Deploying the CloudFormation Stack​

To deploy the CloudFormation stack, use the following command. This command assumes you have AWS CLI configured with the appropriate credentials:

aws cloudformation create-stack --stack-name amazon-ec2-gitlab-runner-demo1 --template-body file://gitlab-runner.yaml --capabilities CAPABILITY_NAMED_IAM

To update the stack, use the following command:

aws cloudformation update-stack --stack-name amazon-ec2-gitlab-runner-demo1 --template-body file://gitlab-runner.yaml --capabilities CAPABILITY_NAMED_IAM

This command will provision a CloudFormation stack similar to table shown below:

Shell-Based Installation Approach​

Rather than using Docker, in your environment, you can use Shell (kernel) for installing GitLab Runner and Terraform directly on the EC2 instances. Using shell rather than container provides the following benefits:

• Simpler Debugging: Direct installation via shell scripts simplifies the debugging process. If something goes wrong, engineers can SSH into the instance and troubleshoot directly rather than dealing with Docker container issues.
• Performance Considerations: Running the runner directly on the EC2 instance reduces the overhead introduced by containerization, potentially improving performance.

Installation Commands​

Below are the key commands used in the shell script for installing GitLab Runner and Terraform:

#!/bin/bash
# Update and install necessary packages
yum update -y
yum install -y amazon-ssm-agent git unzip wget jq

# Install Terraform
wget https://releases.hashicorp.com/terraform/1.0.11/terraform_1.0.11_linux_amd64.zip
unzip terraform_1.0.11_linux_amd64.zip
mv terraform /usr/local/bin/

# Install GitLab Runner
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

# Source GitBash
echo 'export PATH=$PATH:/home/gitlab-runner' >> ~/.bashrc
source ~/.bashrc

Configuration and Usage​

Registering the GitLab Runner​

Once the GitLab Runner is installed, it needs to be registered with your GitLab instance. This process can be automated or done manually. Below is an example of how you can register the runner using the gitlab-runner register command:

gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_REGISTRATION_TOKEN" \
  --executor "shell" \
  --description "GitLab Runner" \
  --tag-list "shell,sgkci/cd" \
  --run-untagged="true" \
  --locked="false"

A simple command:

sudo gitlab-runner register --url https://gitlab.com/ --registration-token <Your registration token>

Example:
sudo gitlab-runner register --url https://gitlab.com/ --registration-token GR1348941Du4BazUzERU5M1m_LeLU

This command registers the GitLab Runner to your GitLab project, allowing it to execute CI/CD pipelines directly on the EC2 instance using the shell executor.

Attaching Runner to GitLab Repo​

Navigate to Repo → Settings → CI/CD. Your runner should show up. Click "Enable for this project," after which the runner should be visible.

Note: To ensure that the runner picks up your job, ensure that the right tag is in place, and you may need to disable the Instance Runners.

🔚 Call to Action​

Choosing the right platform depends on your organizations needs. For more insights, subscribe to our newsletter for insights on cloud computing, tips, and the latest trends in technology. or follow our video series on cloud comparisons.

Interested in having your organization setup on cloud? If yes, please contact us and we'll be more than glad to help you embark on cloud journey.

💬 Comment below:
Which tool is your favorite? What do you want us to review next?