In order to see Thundra data in Elasticsearch there are three steps you will need to complete.
In order to run Thundra Integrator for Elasticsearch AMI, you need to have Thundra Licence Key. Please send an email to firstname.lastname@example.org and we'll turn back to you with your license key which is free for 30-days.
Thundra provides an AWS CloudFormation template to easily set up your integration.
Below is the deployment topology created from the CloudFormation template. It includes:
- The Thundra Receiver App that collects, transforms and aggregates data from the Thundra Lambda agent.
- Filebeat that sends that data to your Elasticsearch.
- A data volume that temporarily stores the files and data collected from the Thundra integrator. This data volume cleaned by the Thundra integrator hourly.
To deploy the Thundra Integration for Elasticsearch AMI:
- Access AWS CloudFormation console
- Click Create Stack
- Select Specify an Amazon S3 template URL and enter https://s3-us-west-2.amazonaws.com/thundra-elasticsearch-staging-dist/thundra-elasticsearch-cf-template.json on which we host the CloudFormation template.
- Click Next
- Fill out the template:
- Enter a Stack Name (Example: thundra-elasticsearch)
- Optional: Enter an EBS Volume Size (Default: 50GBs. Thundra cleans out the data written to this volume hourly. In most cases, 50GBs is enough capacity. However, please feel free to adjust this size for your expected hourly data volumes.
- Select Enable HTTPS to enable or disable SSL with a self-signed certificate
Enable Certificate Validation Using HTTPS
If EnableHTTPS set to ‘True’ please add
thundra_lambda_publish_rest_trustAllCertificates to the environment variables of your AWS lambda function and set to True.
- Enter ElasticSearchUrl
- Enter ElasticSearchPort
- Enter ElasticSearchUsername (optional)
- Enter ElasticSearchPassword ((optional)
- Choose ElasticSearchEnableHttps
- Enter ThundraElasticsearchReceiverElasticIp, this IP can be used to allow Thundra receiver to reach your Elasticsearch instance while sending data.
- Select the appropriate EC2 KeyPair name to enable SSH access to your EC2 instance
- Enter the IP address range that your EC2 instance will have SSH access to
- Enter optional API Key Prefix and/or API Key values. These API keys are used to update the Thundra agent to send data to the Thundra Integrator for Elasticsearch instead of the Thundra Web Console.
If you do not provide an API key or API key prefix the Thundra Integrator for Elasticsearch will accept data without any authentication. We highly discourage this approach as it is not considered best practice.
Enter API Key Prefix. Any Thundra agents that use this API key prefix will be allowed to send data to the Thundra Integrator for Elasticsearch.
Enter allowed API Keys. Any Thundra agent that uses this explicitly set API key will be able to send data to the Thundra Integrator for Elasticsearch. Enter API Key(s) as a comma-separated list.
** IMPORTANT**: Record your API key information!
Write down or record your API key prefix or API keys!!! You will need to enter them into the Thundra agent YML file in Step 3 of the setup.
- Enter the Thundra license key created from the Thundra Web Console
- Enter the port number through which Thundra Integrator for Elasticsearch will receive the Thundra agent data
- Select your EC2 instance type
Below is a list of the parameters and value requirements needed to fill out the CloudFormation template:
||Size of the attached EBS volume in GBs||Numeric value||50|
||To run Thundra receiver over HTTPS with a self-signed certificate, set this parameter to true||true/false||false|
||Keypair name for SSH into EC2 server||AWS EC2 KeyPair values in the region||Empty|
||The IP address range that is allowed to SSH to the EC2 instances||Valid IP range in x.x.x.x/x notation. Use 0.0.0.0/0 for no restrictions.||0.0.0.0/0|
||Prefix of API keys from which the Thundra-Elasticsearch Integrator will receive data. For example, if you set the API key prefix to
||Comma (',') separated list of API keys from which the Thundra Integrator for Elasticsearch will receive data. For example,
||License key for the Thundra Integrator for Elasticsearch can be generated from Thundra Web Console.||Alphanumeric value||Empty|
||Elasticip of your receiver||Numeric value||18.104.22.168|
||Port number for Thundra Integrator for Elasticsearch to receive data over HTTP/HTTPS||Numeric value between 1024 and 65536||8080|
||EC2 instance type||Supported AWS EC2 instance types||d2.xlarge|
||Elasticsearch url||Alphanumeric value||Empty|
||Elasticsearch port||Alphanumeric value||Empty|
||Elasticsearch username||Alphanumeric value||Empty|
||Elasticsearch password||Alphanumeric value||Empty|
||If you want to access Elasticsearch with https, set this value to true||true/false||false|
- When you are finished filling out the template, click
While it is best practice to set tags for your EC2 instance, you are not required to do so to set up this Thundra Integrator for Elasticsearch. These options are specific to CloudFormation. For a detailed explanation see the AWS CloudFormation docs: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html
Nextto move into the Review screen.
- On the Review screen, review your stack details. When ready, click
Createon the lower right corner of the screen to deploy your stack.
Now that the stack creation is complete, you need to send data to the Thundra Integrator for Elasticsearch by updating your publish base URL and API key.
- If this is your first time using Thundra, you will also need to set up the Thundra agents. Otherwise, if you are already have been using Thundra with the web console, these steps will re-direct data to the Thundra Integrator for Elasticsearch instead of the Thundra Web Console.
Set up your Thundra agents if this is your first time using Thundra
Find the endpoint URL for your Thundra Integrator for Elasticsearch in the
Output tab of your CloudFormation stack.
The format should be: <URLofYourThundraElasticsearchIntegratorInstance>/v1.
Now, navigate to your Thundra Lambda environment variable configuration and change the API key and publish base URL.
thundra_apiKey values that you entered during the CloudFormations template set up and
thundra_agent_lambda_report_rest_baseUrl that you obtained from the Output tab in the CloudFormation stack.
Save in the upper right corner of your Thundra Lambda environment variable configuration.
You are now set up for Thundra Integrator for Elasticsearch!
After redeploying your Thundra Lambda with these modifications, you will be able to see the Lambda data on your Kibana UI within approximately 5 minutes.
After successfully deploying theThundra Integrator for Elasticsearch CloudFormation stack, an EC2 instance will be launched with name
Thundra Elasticsearch Receiver.
To manually start and stop your Thundra Integrator for Elasticsearch instance :
- SSH into your EC2 instance with the KeyPair you entered into the CloudFormation template
- Go to
- To start your instance: Run
- To stop your instance: Run