Deployment Integrations
On-premise Integrations
Platform Integrations

Chaos Injection

Span Listeners

Serverless systems are by its nature distributed systems. When you are using serverless architectures which are highly-distributed and event-driven, you should consider being robust in case any of the latencies and errors that can happen with any component. For example; You should be prepared for the case when you get a connection error while trying to reach your Redis cache. Similarly, your system should not collapse when any of the components respond with a latency that slows down the flow. If you are calling a Lambda inside another Lambda make sure that outer one has the bigger timeouts.

It can seem trivial to test the applications in this sense but it will become complicated as your architecture grows. We developed our SpanListener and SpanFilterer APIs for this purpose. When you want to inject chaos to your distributed system, you can use Thundra’s SpanListener API to create the latency or the error that your system can face without changing your code by only playing with the environment variables. The rest of the documentation describes how this can be achieved by Thundra’s Java agent.

What is a span?

Span is a concept that Thundra inherits from the Opentracing. Span is basically an individual unit of work done in your Lambda functions. You can manually create spans using Thundra. However, thanks to the integrations that Thundra have, Thundra creates spans for you automatically if you are using the Traceable annotation or you are using our integrations. Currently, Thundra has following integrations:

  • AWS services (Lambda, DynamoDB, SQS, SNS, Kinesis, Firehose, S3, API Gateway, ...)

  • MySQL

  • PostgreSQL

  • Microsoft SQL Server

  • MariaDB

  • Redis

  • Elasticsearch

  • HTTP

This means that even if you don’t create a span manually, whenever you make a Redis call, HTTP call, DynamoDB call etc. Thundra automatically creates a span for this call and add necessary tags related to that specific operation.

What is a Span Listener?

There are 3 out-of-the-box SpanListener implementation in Thundra Java Agent:

  • ErrorInjectorSpanListener

  • LatencyInjectorSpanListener

  • FilteringSpanListener

ErrorInjectorSpanListener

Parameters:

  • errorMessage: The message to show as the error description when you use ErrorInjectorSpanListener. The default value is “Error injected by Thundra!”.

  • errorType: The type of the error that will be raised. Name of the thrown error will be set to this value. The default value is java.lang.RuntimeException.

  • injectOnFinish: The boolean value that decides whether the error will be injected after the span is finished or before starting. The default value is false.

  • injectPercentage: The integer value that decides percentage for error injection. for example if this value is 10 (means 10%) every 10th span event will be an error, the default value is 100.

ErrorInjectorSpanListener
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
...
// Create span listener
ErrorInjectorSpanListener spanListener =
new ErrorInjectorSpanListener(
false, // don't inject on finish but on start before the actual operation
java.lang.IllegalAccessException.class, // type of the exception to be thrown
"No way!", // the error message
50 // inject error at 50% percentage
);
// Register span listener
TraceSupport.registerSpanListener(spanListener);

LatencyInjectorSpanListener

Parameters:

  • delay: The integer value that decides the amount of latency that will be injected to the spans in milliseconds. The default value is 100 ms.

  • randomizeDelay: The boolean value that enables randomization amounts of injected delays. For example, if the delay is 100 ms and randomization is enabled, a random number is generated between 1 and 100 as the amount of each injected latency in milliseconds. The default value is false.

  • injectOnFinish: The boolean value that decides whether the latency will be injected after the span is finished or before starting. The default value is false.

LatencyInjectorSpanListener
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.span.impl.LatencyInjectorSpanListener;
...
// Create span listener
LatencyInjectorSpanListener spanListener =
new LatencyInjectorSpanListener(
false, // don't inject on finish but on start before the actual operation
50, // amount of delay to be injected in milliseconds
true // randomize amount of injected delays between 1 and 50 (configured delay)
);
// Register span listener
TraceSupport.registerSpanListener(spanListener);

FilteringSpanListener

Parameters:

  • listener: The SpanListener implementation which is notified after FilteringSpanListener filters the span using its filterer parameter if the span is accepted by the filterer. There is no default value is this parameter is mandatory`.

  • filterer: Filterer is the main object that decides whether the given span will be delegated to the listener parameter or not. It is basically an object that has an accept method. Before delegating the given span to the listener parameter, FilteringSpanListener first passes the given span to the filterer object if the filterer’s accept method returns true, then the given span is passed to the listener parameter. Otherwise, the given span is not passed to the listener. There is no default value is this parameter is mandatory`.

For example,

FilteringSpanListener
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
import io.thundra.agent.trace.span.impl.FilteringSpanListener;
import io.thundra.agent.trace.instrument.integrations.redis.RedisTags;
...
// Create error injector span listener
ErrorInjectorSpanListener listener =
new ErrorInjectorSpanListener(
false, // don't inject on finish but on start before the actual operation
java.lang.IllegalAccessException.class, // type of the exception to be thrown
"No way!", // the error message
50 // inject error at 50% percentage
);
// Create filterer to decide which spans will be listened
FilteringSpanListener.SpanFilterer filterer =
new FilteringSpanListener.SpanFilterer() {
@Override
public boolean accept(ThundraSpan span) {
return "Redis".equals(span.getClassName()) // Targeting only Redis operations
&&
"GET".equalsIgnoreCase(span.getTag(RedisTags.COMMAND)); // Targeting only Redis "GET" commands
}
};
// Create filtering span listener
FilteringSpanListener filteringSpanListener =
new FilteringSpanListener(
listener,
filterer
);
// Register span listener
TraceSupport.registerSpanListener(filteringSpanListener);

or the same configuration can be done over SpanFilter:

FilteringSpanListener
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.span.impl.ErrorInjectorSpanListener;
import io.thundra.agent.trace.span.impl.FilteringSpanListener;
import io.thundra.agent.trace.instrument.integrations.redis.RedisTags;
import java.util.Arrays;
import java.util.HashMap;
...
// Create error injector span listener
ErrorInjectorSpanListener listener =
new ErrorInjectorSpanListener(
false, // don't inject on finish but on start before the actual operation
java.lang.IllegalAccessException.class, // type of the exception to be thrown
"No way!", // the error message
50 // inject error at 50% percentage
);
// Create filterer to decide which spans will be listened
FilteringSpanListener.SpanFilter filter =
new FilteringSpanListener.SpanFilter(
null, // "domainName" is not checked
"Redis", // Targeting only Redis operations
null, // "operationName" is not checked
new HashMap() {{
put(RedisTags.COMMAND, "GET"); // Targeting only Redis "GET" commands
}}
);
// Create filtering span listener
FilteringSpanListener filteringSpanListener =
new FilteringSpanListener(
listener,
Arrays.asList(filter)
);
// Register span listener
TraceSupport.registerSpanListener(filteringSpanListener);

Configuration using environment variables: In order to create register span listener using an environment variable, you should set thundra_agent_lambda_trace_span_listenerConfig prefixed (multiple listener definitions can have different suffix) environment variable in JSON format. The basic structure is like to following:

Basic structure of a span listener created using the environment variable
thundra_agent_lambda_trace_span_listenerConfig...: '{"type":"<span-listener-name>","config":{[<arguments>]}'

where

  • <span-listener-name> can be full classname of the SpanListener implementation of one of the following predefined ones (ErrorInjectorSpanListener, LatencyInjectorSpanListener or FilteringSpanListener).

  • <arguments> are list of objects which have key-value pairs.

Here is an example configuration that creates the same filtering span listener that we create programmatically above:

A FilteringSpanListener configured using the environment variable
thundra_agent_lambda_trace_span_listenerConfig:'{"type":"FilteringSpanListener","config":{"listener":{"type":"ErrorInjectorSpanListener","config":{"errorType":"java.lang.IllegalAccessException","errorMessage":"Noway!","injectPercentage":50}},"filters":[{"className":"Redis","tags":{"redis.command":"GET"}}]}}'

There are a couple of points to note here:

  • Using the "type":"LatencyInjectorSpanListener", we state the type of the listener that FilteringSpanListener is going to use. You can use LatencyInjectorSpanListener too when you need to inject latency.

  • Key-value pairs under the config property are passed to the wrapped span listener, which is specified by type property (which is ErrorInjectorSpanListener in this case), as parameters.

  • Key-value pairs under the config property are used to define individual filters. For example, {"className":"Redis","tags": {"redis.command":"GET"}} corresponds to the following filter:

SpanFilter
FilteringSpanListener.SpanFilter filter =
new FilteringSpanListener.SpanFilter(
null, // "domainName" is not checked
"Redis", // Targeting only "Redis" operations
null, // "operationName" is not checked
new HashMap() {{
put(RedisTags.COMMAND, "GET"); // Targeting only Redis "GET" commands
}}
);

Resulting filtering span listener would first filter the spans by looking at the className and the tags of the spans. If the given span has a different className then the Redis or the given span does not have the tag redis.command with the value GET, then the span will be rejected and won’t be sent to the listener. If it does have both properties then the filterer accepts the span and FilteringSpanListener propagates the span to its listener parameter (which is a ErrorInjectorSpanListener) in this case.

SpanFilter

Parameters:

  • domainName: Domain name value to compare with the given span’s domain name. The default value is null

  • className: Class name value to compare with the given span’s class name. The default value is null

  • operationName: Operation name value to compare with the given span’s operation name. The default value is null

  • tags: A dictionary that contains key-value tag pairs. SpanFilter checks that each of the key-value pairs in this dictionary also exists in the given span’s tags. If at least one of the key-value pairs not exists in the span’s tags or exist with a different value then the SpanFilter rejects the span. The default value is null