2 posts tagged with "Terraform"

Introduction​

This document outlines the steps taken to deploy and configure GitLab Runners, including the installation of Terraform, ensuring that the application team can focus solely on writing pipelines.

Architecture​

The following diagram displays the solution architecture.

AWS CloudFormation is used to create the infrastructure hosting the GitLab Runner. The main steps are as follows:

1. The user runs a deploy script to deploy the CloudFormation template. The template is parameterized, and the parameters are defined in a properties file. The properties file specifies the infrastructure configuration and the environment in which to deploy the template.
2. The deploy script calls CloudFormation CreateStack API to create a GitLab Runner stack in the specified environment.
3. During stack creation, an EC2 autoscaling group is created with the desired number of EC2 instances. Each instance is launched via a launch template created with values from the properties file. An IAM role is created and attached to the EC2 instance, containing permissions required for the GitLab Runner to execute pipeline jobs. A lifecycle hook is attached to the autoscaling group on instance termination events, ensuring graceful instance termination.
4. During instance launch, GitLab Runner will be configured and installed. Terraform, Git, and other software will also be installed as needed.
5. The user may repeat the same steps to deploy GitLab Runner into another environment.

Infrastructure Setup with CloudFormation​

Customizing the CloudFormation Template​

The initial step in deploying GitLab Runners involved setting up the infrastructure using AWS CloudFormation. The standard CloudFormation template was customized to fit the unique requirements of the environment.

CloudFormation Template Location: GitLab Runner Template

CloudFormation Template Location: GitLab Runner Scaling Group / Cluster Template

For any automation requirement or issues, please reach out to us Contact Us

Parameters used:

Deploying the CloudFormation Stack​

To deploy the CloudFormation stack, use the following command. This command assumes you have AWS CLI configured with the appropriate credentials:

aws cloudformation create-stack --stack-name amazon-ec2-gitlab-runner-demo1 --template-body file://gitlab-runner.yaml --capabilities CAPABILITY_NAMED_IAM

To update the stack, use the following command:

aws cloudformation update-stack --stack-name amazon-ec2-gitlab-runner-demo1 --template-body file://gitlab-runner.yaml --capabilities CAPABILITY_NAMED_IAM

This command will provision a CloudFormation stack similar to table shown below:

Shell-Based Installation Approach​

Rather than using Docker, in your environment, you can use Shell (kernel) for installing GitLab Runner and Terraform directly on the EC2 instances. Using shell rather than container provides the following benefits:

• Simpler Debugging: Direct installation via shell scripts simplifies the debugging process. If something goes wrong, engineers can SSH into the instance and troubleshoot directly rather than dealing with Docker container issues.
• Performance Considerations: Running the runner directly on the EC2 instance reduces the overhead introduced by containerization, potentially improving performance.

Installation Commands​

Below are the key commands used in the shell script for installing GitLab Runner and Terraform:

#!/bin/bash
# Update and install necessary packages
yum update -y
yum install -y amazon-ssm-agent git unzip wget jq

# Install Terraform
wget https://releases.hashicorp.com/terraform/1.0.11/terraform_1.0.11_linux_amd64.zip
unzip terraform_1.0.11_linux_amd64.zip
mv terraform /usr/local/bin/

# Install GitLab Runner
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

# Source GitBash
echo 'export PATH=$PATH:/home/gitlab-runner' >> ~/.bashrc
source ~/.bashrc

Configuration and Usage​

Registering the GitLab Runner​

Once the GitLab Runner is installed, it needs to be registered with your GitLab instance. This process can be automated or done manually. Below is an example of how you can register the runner using the gitlab-runner register command:

gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_REGISTRATION_TOKEN" \
  --executor "shell" \
  --description "GitLab Runner" \
  --tag-list "shell,sgkci/cd" \
  --run-untagged="true" \
  --locked="false"

A simple command:

sudo gitlab-runner register --url https://gitlab.com/ --registration-token <Your registration token>

Example:
sudo gitlab-runner register --url https://gitlab.com/ --registration-token GR1348941Du4BazUzERU5M1m_LeLU

This command registers the GitLab Runner to your GitLab project, allowing it to execute CI/CD pipelines directly on the EC2 instance using the shell executor.

Attaching Runner to GitLab Repo​

Navigate to Repo → Settings → CI/CD. Your runner should show up. Click "Enable for this project," after which the runner should be visible.

Note: To ensure that the runner picks up your job, ensure that the right tag is in place, and you may need to disable the Instance Runners.

🔚 Call to Action​

Choosing the right platform depends on your organizations needs. For more insights, subscribe to our newsletter for insights on cloud computing, tips, and the latest trends in technology. or follow our video series on cloud comparisons.

Interested in having your organization setup on cloud? If yes, please contact us and we'll be more than glad to help you embark on cloud journey.

💬 Comment below:
Which tool is your favorite? What do you want us to review next?

To become a good cloud architect it's important to understand the essential pillars that support a well-architected framework. This framework helps in designing, deploying, and maintaining cloud applications efficiently. Here are some of the key pillars and insights from our experience at Arena Technologies.

1. Operational Excellence​

Operational excellence involves running and monitoring systems to deliver business value and continuously improve processes and procedures. It’s crucial to have integration, security, incident monitoring, and automation in place.

Technologies to Learn:

• Monitoring and Logging: AWS CloudWatch / Azure Monitor / Google Stackdriver
• CI/CD: Jenkins / GitLab CI / CircleCI
• Infrastructure as Code: Terraform / CloudFormation / ARM Templates

2. Security​

Security is the foundation of any cloud architecture. It involves infrastructure security, network security, application security, and DevSecOps practices. Security should be considered from day zero, even before starting the project.

Technologies to Learn:

• Identity and Access Management: AWS IAM / Azure AD / Google IAM
• Key Management: AWS KMS / Azure Key Vault / Google Cloud KMS
• Application Security: OWASP Tools / Snyk

3. Reliability​

Reliability ensures a workload performs its intended function correctly and consistently. This includes planning for disaster recovery, high availability, and redundancy.

Technologies to Learn:

• Traffic Routing: AWS Route 53 / Azure Traffic Manager / Google Cloud DNS
• Database Redundancy: AWS RDS Multi-AZ / Azure SQL Database Geo-Replication / Google Cloud Spanner
• Data Backup and Disaster Recovery: AWS Backup / Azure Backup / Google Cloud Backup

4. Performance Efficiency​

Performance efficiency is about using IT and computing resources efficiently. This includes selecting the right instance types, optimizing storage, and ensuring that your application scales to meet demand.

Technologies to Learn:

• Scaling Compute Resources: AWS Auto Scaling / Azure VM Scale Sets / Google Cloud Autoscaler
• Scalable Storage Solutions: AWS S3 / Azure Blob Storage / Google Cloud Storage
• Serverless Computing: AWS Lambda / Azure Functions / Google Cloud Functions

5. Cost Optimization​

Cost optimization involves controlling where the money is being spent, selecting the most appropriate and right number of resource types, and scaling to meet business needs without overspending.

Technologies to Learn:

• Cost Monitoring and Management: AWS Cost Explorer / Azure Cost Management / Google Cloud Pricing Calculator
• Setting and Monitoring Budgets: AWS Budgets / Azure Budgets / Google Cloud Budgets
• Optimizing Costs with Long-term Commitments: Spot Instances / Reserved Instances / Savings Plans

6. Sustainability​

Sustainability in cloud computing involves designing solutions that reduce carbon footprint and manage resources responsibly.

Technologies to Learn:

• Sustainability Practices: AWS Sustainability Practices / Azure Sustainability Practices / Google Sustainability Practices
• Energy-efficient Algorithms: To optimize compute usage

Important Aspects of Cloud Architecture​

Architecture​

A solid architecture is crucial for any cloud setup. Unlike traditional on-premises setups, cloud architecture must be designed with scalability and efficiency in mind. Common architectural patterns include microservices, service-oriented architecture (SOA), and data pipeline architectures.

Technologies to Learn:

• Container Orchestration: Kubernetes / Amazon EKS / Azure AKS / Google GKE
• Container Management: AWS ECS / Azure Container Instances / Google Cloud Run
• Service Mesh: Istio / Linkerd

Automation​

Automation is essential in cloud environments. Tools like Terraform for infrastructure as code (IaC) and continuous integration/continuous deployment (CI/CD) pipelines ensure that your infrastructure and deployments are consistent, repeatable, and scalable.

Technologies to Learn:

• Infrastructure as Code: Terraform / CloudFormation / ARM Templates
• CI/CD Pipelines: Jenkins / GitLab CI / CircleCI
• Configuration Management: Ansible / Chef / Puppet

Application and Data​

Understanding application architecture and data management is crucial. Depending on the application type—whether it’s a web service, big data application, or something else—the architectural and technological choices will vary. It is important to choose the right databases and data management tools based on your specific needs.

Technologies to Learn:

• Relational Databases: AWS RDS / Azure SQL Database / Google Cloud SQL
• NoSQL Databases: AWS DynamoDB / Azure Cosmos DB / Google Cloud Firestore
• Real-time Data Streaming: Apache Kafka / AWS Kinesis / Azure Event Hubs / Google Pub/Sub

Non-Functional Requirements​

Non-functional requirements (NFRs) are often overlooked but are critical to the success of any cloud project. These include:

• Performance: How well the system performs under load.
• Scalability: The ability to scale up or down as needed.
• High Availability: Ensuring the system is operational at all times.
• Disaster Recovery: Planning for system recovery in case of failures.

Practical Tips for Aspiring Cloud Architects​

• Learn Multiple Architectural Patterns: Familiarize yourself with different architecture styles and understand when to use each.
• Understand Security Practices: Security must be integrated into every part of your architecture.
• Embrace Automation: Use tools like Terraform and CI/CD pipelines to automate as much as possible.
• Focus on Cost Management: Keep an eye on costs from the beginning to avoid unexpected expenses.
• Stay Updated: Cloud technologies evolve rapidly, so continuous learning is key.

Technologies to Learn:

• Architectural Best Practices: AWS Well-Architected Tool / Azure Well-Architected Review / Google Cloud Architecture Framework
• Optimizing and Improving Cloud Environments: AWS Trusted Advisor / Azure Advisor / Google Cloud Advisor

Conclusion​

Being a good cloud architect requires a blend of technical knowledge, practical experience, and an understanding of the broader business context. By focusing on the pillars of a well-architected framework and considering both functional and non-functional requirements, you can design efficient, scalable, and secure cloud solutions.