Chaos Engineering with Thundra

While developing an application it is inevitable to have bottlenecks and failures in your business logic or infrastructure. It is important to find these issues before your application goes to production and fix them. Since you don't want to affect users with failures, you should test your system before it causes much more damages in production.

Testing your system with controlled experiments to identify behaviour of system when a fallback is happened is named as Chaos Engineering.

Thundra provides full observability for your serverless system. You can list and filter your function with detailed queries. If you want to have a deeper insight, click on a function and display details of it. Invocations, metrics and performance analysis of your function is displayed in Function Details page.

Thundra helps you to see what happened in an invocation, how much time spent on it by drilling into Invocation Details. In a problem-free scenario, you can see everything is okay in this invocation. However, what happens if something goes wrong such as problem about writing to DynamoDB or latency issues? At this point, Chaos Injection feature of Thundra is what you are looking for!

You can inject errors to your Lambda functions with Thundra to simulate failures in your serverless system. Thundra supports Chaos Injection in Java, NodeJS and Python for now.

What is a Span?

Span is a concept that Thundra inherits from the Opentracing. Span is basically an individual unit of work done in your Lambda functions. You can create your custom spans using Thundra if you want. However, if you use automated instrumentation for your functions, Thundra creates spans for you. In Thundra console, you can display these spans under invocation's trace chart. Thundra supports following integrations for spans:

  • Botocore

  • Dynamodb

  • SQS

  • SNS

  • Kinesis

  • Firehose

  • S3

  • Lambda

  • MySQL

  • PostgreSQL

  • Redis

  • Requests (HTTP)

This means that even if you don’t create a span manually, whenever you make a Redis call, HTTP call, DynamoDB call etc. Thundra automatically creates a span for this call and add necessary tags related to that specific operation.

What is Span Listener?

Span listeners are the classes that inherit from ThundraSpanListener in order to create spans in your Lambda function usingon_span_started(span), on_span_finished(span) methods. You can create span using span listeners and perform custom operations in lifecycle of this span. Span listener can be used to perform fallbacks in your functions. However, Thundra provides you 3 ready to use span listener to be used for Chaos Injection:

You can add these listeners to your code manually to see behaviour of your serverless system during a problem.

Below you can find how to inject chaos to your serverless environment.

How to Inject Chaos?

For an example usage and detailed view of Chaos Injection in Thundra you can read our blog post!

For configuring chaos engineering in each environment explore our documentation pages: Java, Python, NodeJS.