Deployment Integrations
On-premise Integrations
Platform Integrations

Chaos Injection

Span Listeners

Span listeners are the classes that inherit from the ThundraSpanListener abstract base class which have on_span_started(span), on_span_finished(span) methods. When you register a span listener to Thundra, Thundra calls this span listener’s on_span_started(span) and on_span_finished(span) methods whenever a span is started and finished. This way, you can perform custom operations using a span listener during the lifecycle of a span.

Thundra provides following span listener implementations:

  • ErrorInjectorSpanListener

  • LatencyInjectorSpanListener

  • FilteringSpanListener

ErrorInjectorSpanListener

Parameters:

  • error_message: The message to show as the error description when you use ErrorInjectorSpanListener. The default value is “Error injected by Thundra!”.

  • error_type: The type of the error that will be raised. Use classes that inherit from the BaseException class for this parameter. Do not pass objects, but the class itself (eg. pass NameError instead of NameError( ) ). The default value is Exception.

  • inject_on_finish: Boolean value that decides whether the error will be injected after the span is finished or before starting. The default value is False.

  • add_info_tags: Span listener adds information tags to the spans when the add_info_tags parameter is set to True. These tags contain the span listeners parameter names and values. So that in Thundra console, you can see the span listener parameters on the corresponding span’s tags section. The default value is True.

Programmatic configuration of an ErrorInjectorSpanListener
import redis
from thundra.plugins.trace import trace_support
from thundra.listeners import ErrorInjectorSpanListener
# Create the listener
error_sl = ErrorInjectorSpanListener(
error_type=redis.ConnectionError,
error_message="This is an injected error!"
inject_on_finish=True
)
# Register the listener to use it
trace_support.register_span_listener(error_sl)

LatencyInjectorSpanListener

Parameters:

  • delay: The integer value that decides the latency that will be injected to the spans in milliseconds. The default value is 0.

  • variation: When the distribution parameter is set to uniform, the variationparameter decides the range from which the injected latency value will be sampled. For example, if the delay is 500 and the variation is 150, then the amount of latency that will be injected to the span will be a uniform random variable between the values [500-150, 500+150] = [350, 650]. The default value is 0.

  • sigma: When the distribution parameter is set to normal, sigma (standard deviation) parameter decides the range from which the injected latency value will be sampled. If the distribution parameter is normal, the amount of latency that will be injected to the span will be a normally distributed random variable with a standard deviation is equal to the sigma parameter and a mean value is equal to the delay parameter. The default value is 0.

  • distribution: distribution parameter decides which distribution to use to when sampling the amount of latency injected. If it is set to uniform value, then the span listener will use the variation and delay parameters to decide the amount of the latency. If it is set to normal, then the span listener will use the sigma parameter and the delay parameter to decide the amount of the latency. If it is set to a value other then the uniform and normal, span listener will behave like it is set to uniform and decide the amount of latency accordingly. The default value is uniform.

Programmatic configuration of a LatencyInjectorSpanListener
from thundra.plugins.trace import trace_support
from thundra.listeners import LatencyInjectorSpanListener
# Create the listener
latency_sl = LatencyInjectorSpanListener(
delay=2000,
sigma=1000,
distribution='normal',
)
# Register it to use it
trace_support.register_span_listener(latency_sl)

FilteringSpanListener

Parameters:

  • listener: After FilteringSpanListener filters the span using its filterer parameter, if the span is accepted by the filterer, then it is sent to the span listener given by the listener parameter. The default value is None.

  • filterer: Filterer is the main object that decides whether the given span will be delegated to the listener parameter or not. It is basically an object that has an accept method. Before delegating the given span to the listener parameter, FilteringSpanListener first passes the given span to the filterer object if the filterer’s accept method returns True, then the given span is passed to the listenerparameter. Otherwise, the given span is not passed to the listener. The default value is None.

Programmatic configuration of a FilteringSpanListener
from thundra.listeners import FilteringSpanListener
from thundra.listeners import LatencyInjectorSpanListener
from thundra.plugins.trace import trace_support
from thundra.listeners.thundra_span_filterer import ThundraSpanFilterer,ThundraSpanFilter
# Create main listener
latency_sl = LatencyInjectorSpanListener(
delay=2000,
sigma=1000,
distribution='normal',
)
# Create a filter
f1 = ThundraSpanFilter(
class_name='AWS-Lambda',
tags={'aws.lambda.name': 'upstream-lambda'}
)
# Create a filterer that uses a list of filters
filterer = ThundraSpanFilterer(span_filters=[f1])
# Create filtering span listener that takes the listener and the filterer
filtering_sl = FilteringSpanListener(
listener=latency_sl,
filterer=filterer,
)
# Register the listener to use it
trace_support.register_span_listener(filtering_sl)

Configuration using environment variables: In order to create a filtering span listener using an environment variable, you should set thundra_agent_lambda_trace_span_listener environment variable. The basic structure is like to following:

Basic structrure of a filtering span listener created using the environment variable
thundra_agent_lambda_trace_span_listener: FilteringSpanListener[arguments]

Here is an example configuration that creates the same filtering span listener that we create programmatically above:

A FilteringSpanListener configured using the environment variable
thundra_agent_lambda_trace_span_listener: FilteringSpanListener[listener=LatencyInjectorSpanListener,config.delay=2000,config.sigma=1000,config.distribution=normal,filter1.className=AWS-Lambda,filter1.tag.aws.lambda.name=upstream-lambda]

There are a couple of points to note here:

  • Using the listener=LatencyInjectorSpanListener, we state the type of the listener that FilteringSpanListener is going to use. You can use ErrorInjectorSpanListener too when you need to inject an error.

  • Each argument starting with config. prefix is going to be passed to the listenerparameter (which is LatencyInjectorSpanListener in this case) as parameters. It has a structure like this: config.parameterName. Note that we are using camel case whenever we need it. For example, if you used ErrorInjectorSpanListener as the listenerargument, then you would pass the error_message parameter like this: config.errorMessage=”Hello world!”

  • Each argument that starts with the filter keyword creates a filter. Any other character that follows the filter keyword is basically id of that filter.

  • For example, filter1.className=AWS-Lambda, filter1.tag.aws.lambda.name=upstream-lambda. these two arguments correspond to the following filter:

Corresponding filter1 created using the environment variable
filter1 = {
class_name="AWS-Lambda",
tags={
'aws.lambda.name': 'upstream-lambda'
}
}

Resulting filtering span listener would first filter the spans by looking at the class_nameand the tags of the spans. If the given span has a different class_name then the “AWS-Lambda” or the given span does not have the tag aws.lambda.name with the value “upstream-lambda”, then the span will be rejected and won’t be sent to the listener. If it does have both properties then the filterer accepts the span and FilteringSpanListener propagates the span to its listener parameter(which is a LatencyInjectorSpanListener) in this case.

ThundraSpanFilterer

Parameters:

  • span_filters: A list of span filters. Whenever a FilteringSpanListener calls the ThundraSpanFilterer’s accept method to decide whether or not to accept the given span, ThundraSpanFilterer internally calls the accept(span) method of each of the filters in the span_filters. If at least one of the filters in the span_filters accepts the span, or the span_filters list is empty, then the given span is accepted and returned True. Otherwise, ThundraSpanFilterer’s accept method returns False. The default value is []

ThundraSpanFilter

Parameters:

  • domain_name: Domain name value to compare with the given span’s domain name. The default value is None

  • class_name: Class name value to compare with the given span’s class name. The default value is None

  • operation_name: Operation name value to compare with the given span’s operation name. The default value is None

  • tags: A dictionary that contains key-value tag pairs. ThundraSpanFilter checks that each of the key-value pairs in this dictionary also exists in the given span’s tags. If at least one of the key-value pairs not exists in the span’s tags or exist with a different value then the ThundraSpanFilter rejects the span. The default value is None