Automate Your OpenSearch/Elasticsearch Backups with S3 and Lambda: A Complete Guide
In the world of data management and cloud computing, ensuring data security through regular backups is crucial. OpenSearch and Elasticsearch provide robust mechanisms to back up data using snapshots, offering several approaches to cater to different operational needs. This blog post will walk you through setting up and managing snapshots using AWS, with detailed steps for both beginners and advanced users.
Introduction to Snapshots in OpenSearch and Elasticsearch
Snapshots are point-in-time backups of your OpenSearch or Elasticsearch data. By taking snapshots at regular intervals, you can ensure your data is always backed up, which is especially important in production environments. Snapshots can be scheduled to run automatically, whether hourly, daily, or at another preferred frequency, making it easy to maintain a stable backup routine.
Setting Up an OpenSearch Cluster on AWS
Before diving into snapshot creation, its essential to set up an OpenSearch cluster. Here is how:
- AWS Console Access: Begin by logging into your AWS Console and navigating to OpenSearch.
- Cluster Creation: Create a new OpenSearch domain (essentially your cluster) using the "Easy Create" option. This option simplifies the setup process, especially for demonstration or learning purposes.
- Instance Selection: For this setup, select a lower instance size if you are only exploring OpenSearch features and dont require high memory or compute power. For this demo, an m5.large instance with minimal nodes is sufficient.
Configuring the Cluster
When configuring the cluster, adjust the settings according to your requirements:
- Memory and Storage: Set minimal storage (e.g., 10 GB) to avoid unnecessary costs.
- Node Count: Choose a single-node setup if you are only testing the system.
- Access Control: For simplicity, keep public access open, though in production, you should configure a VPC and control access strictly.
Snapshot Architecture: AWS Lambda and S3 Buckets
AWS provides a serverless approach to managing snapshots via Lambda and S3 buckets. Here is the basic setup:
- Create an S3 Bucket: This bucket will store your OpenSearch snapshots.
- Lambda Function for Snapshot Automation: Use AWS Lambda to automate the snapshot process. Configure the Lambda function to run daily or at a frequency of your choice, ensuring backups are consistent and reliable.
Writing the Lambda Code
For the Lambda function, Python is a convenient choice, but you can choose other languages as well. The Lambda function will connect to OpenSearch, initiate a snapshot, and store it in the S3 bucket. Here is a simple breakdown of the code structure:
import boto3, os, time
import requests
from requests_aws4auth import AWS4Auth
from datetime import datetime
import logging
from requests.adapters import HTTPAdapter, Retry
# Set the global variables
# include https:// and trailing /
host = str(os.getenv('host'))
region = str(os.getenv('region','eu'))
s3Bucket = str(os.getenv('s3Bucket'))
s3_base_path = str(os.getenv('s3_base_path','daily'))
s3RepoName = str(os.getenv('s3RepoName'))
roleArn = str(os.getenv('roleArn'))
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
s3 = boto3.client('s3')
def lambda_handler(event, context):
datestamp = datetime.now().strftime('%Y-%m-%dt%H:%M:%S')
# Register repository
# the Elasticsearch API endpoint
path = '_snapshot/'+s3RepoName
url = host + path
snapshotName = 'snapshot-'+datestamp
# Setting for us-east-1. Comment below if another region
payload = {
"type": "s3",
"settings": {
"bucket": s3Bucket,
"base_path": s3_base_path,
"endpoint": "s3.amazonaws.com",
"role_arn": roleArn
}
}
headers = {"Content-Type": "application/json"}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)
# Take snapshot - Even though this looks similar to above, but this code is required to take snapshot.
# Snapshot to take with datestamp concatanetated - this creates separate snapshots
path = '_snapshot/'+s3RepoName+'/'+snapshotName
url = host + path
string = snapshotName
bucket_name = s3Bucket
s3 = boto3.resource("s3")
s3.Bucket(bucket_name).put_object(Key=s3_path, Body=string)
print(f"Created {s3_path}")
### Text File copying ends here
while True:
response = requests.put(url, auth=awsauth)
status_code = response.status_code
print("status_code == "+str(status_code))
if status_code >= 500:
# Hope it won't 500 a little later
print("5xx thrown. Sleeping for 200 seconds.. zzzz...")
time.sleep(200)
else:
print(f"Snapshot {snapshotName} successfully taken")
break
print(r.text)
- Snapshot API Call: The code uses the OpenSearch API to trigger snapshot creation. You can customize the frequency to take snapshots.
- Error Handling: In scenarios where snapshots take long, retries and error handling are implemented in to manage API call failures.
- Permissions Setup: Grant your Lambda function the necessary permissions to access OpenSearch and the S3 bucket. This includes setting up roles and policies in AWS Identity and Access Management (IAM).
- Invocation Permissions: Lambda function will need to have role that allows access to OpenSearch domain. The role should allow Lambda to upload snapshots to s3 bucket:
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::<BUCKET-NAME>/*"
]
}
Creating an AWS Lambda Layer for the Requests Library
To create a custom AWS Lambda layer specifically for the requests library, follow these steps. This guide will help you set up the requests package as a Lambda layer so it can be reused across multiple Lambda functions.
Follow these steps to create a custom AWS Lambda layer that includes the requests library.
Step 1: Prepare the Requests Dependency
Since Lambda layers require dependencies to be packaged separately, we need to install the requests library in a specific structure.
Set Up a Local Directory
Create a folder structure for installing the dependency.
mkdir requests-layer
cd requests-layer
mkdir python
Install the Requests Library
Use pip to install requests into the python folder:
pip install requests -t python/
Verify Installation
Check that the python directory contains the installed requests package:
ls python
You should see a folder named requests, confirming that the package was installed successfully.
Step 2: Create a Zip Archive of the Layer
After installing the dependencies, zip the python directory:
zip -r requests-layer.zip python
This creates a requests-layer.zip file that you will upload as a Lambda layer.
Step 3: Upload the Layer to AWS Lambda
- Open the AWS Lambda Console
- Go to the AWS Lambda Console.
- Create a New Layer
- Select Layers from the left-hand navigation.
- Click Create layer.
- Configure the Layer
- Name: Provide a name like requests-layer.
- Description: Optionally, describe the purpose of the layer.
- Upload the .zip file: Choose the requests-layer.zip file you created.
- Compatible runtimes: Choose the runtime(s) that match your Lambda function, such as Python 3.8, Python 3.9, or Python 3.10. 11.Create the Layer
- Click Create to upload the layer.
Step 4: Add the Layer to Your Lambda Function
1.Open Your Lambda Function 2.In the Lambda Console, open the Lambda function where you want to use requests. 3.Add the Layer 4.In the Layers section, click Add a layer. 5.Select Custom layers and choose the requests-layer. 6.Select the specific version (if there are multiple versions). 7.Click Add.
OpenSearch Dashboard Configuration
The OpenSearch Dashboard (formerly Kibana) is your go-to for managing and monitoring OpenSearch. Here is how to set up your snapshot role in the dashboard:
- Access the Dashboard: Navigate to the OpenSearch Dashboard using the provided domain link.
- Role Setup: Go to the security settings and create a new role for managing snapshots. Grant this role permissions to access the necessary indices and S3 bucket. Following is the role that needs to be created:
Trust Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"opensearch.amazonaws.com",
"es.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
Role Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": [
"arn:aws:s3:::BUCKET-NAME"
]
},
{
"Effect": "Allow",
"Action": [
"s3:*Object"
],
"Resource": [
"arn:aws:s3:::BUCKET-NAME/*"
]
},
{
"Sid": "ESaccess",
"Effect": "Allow",
"Action": [
"es:*"
],
"Resource": [
"arn:aws:es:eu-west-2:<ACCOUNT-NUMBER>:domain/*"
]
}
]
}
- Mapping the Role: Map the new role to your Lambda functions IAM role to ensure seamless access.
Setting Up Snapshot Policies in the Dashboard
The OpenSearch Dashboard allows you to create policies for managing snapshots, making it easy to define backup schedules and retention periods. Here is how:
- Policy Configuration: Define your backup frequency (daily, weekly, etc.) and the retention period for each snapshot.
- Retention Period: Set the maximum number of snapshots to keep, ensuring that old snapshots are automatically deleted to save space.
- Notification Channel: You can set up notifications (e.g., via Amazon SNS) to alert you if a snapshot operation fails.
Testing and Troubleshooting Your Snapshot Setup
Once your setup is complete, it is time to test it:
- Run a Test Snapshot: Trigger your Lambda function manually and check your S3 bucket for the snapshot data.
- Verify Permissions: If you encounter errors, check your IAM roles and permissions. Snapshot failures often occur due to insufficient permissions, so make sure both the OpenSearch and S3 roles are configured correctly.
- Monitor Logs: Use CloudWatch logs to review the execution of your Lambda function, which will help in troubleshooting any issues that arise.
Disaster Recovery and Restoring Snapshots
In the unfortunate event of data loss or a disaster, restoring your data from snapshots is straightforward. Here is a simple guide:
- New Cluster Setup: If your original cluster is lost, create a new OpenSearch domain.
- Restore Snapshot: Use the OpenSearch API to restore the snapshot from your S3 bucket.
- Cluster Health Check: Once restored, check the health of your cluster and validate that your data is fully recovered.
Conclusion
Using AWS Lambda and S3 for snapshot management in OpenSearch provides a scalable and cost-effective solution for data backup and recovery. By setting up automated snapshots, you can ensure that your data is consistently backed up without manual intervention. With the additional security and monitoring tools provided by AWS, maintaining the integrity and availability of your OpenSearch data becomes a manageable task.
Explore the various options within AWS and OpenSearch to find the configuration that best fits your environment. And as always, remember to test your setup thoroughly to prevent unexpected issues down the line.
For more tips on OpenSearch, AWS, and other cloud solutions, subscribe to our newsletter and stay up-to-date with the latest in cloud technology! Ready to take your cloud infrastructure to the next level? Please reach out to us
