Thundra

Thundra: Serverless Observability for AWS Lambda

The black box nature of AWS Lambda and other serverless environments means that identifying and fixing performance issues is difficult and time-consuming. Built for straightforward debugging, monitoring, and observability, Thundra provides deep insight into your entire serverless environment. Thundra collects and correlates all your metrics, logs, and traces, allowing you to quickly identify problematic invocations and also analyzes external services associated with that function. With Thundra’s zero overhead and automated instrumentation capabilities, your developers are free to write code without worrying about bulking up their Lambdas or wasting time on chasing black box problems.

Get Started    Discussions

How to Instrument Java Functions

Instrumentation of your Java functions lets you perform in-depth monitoring of the behavior of your functions as they are invoked and executed. This means that you can visualize how your functions interact with one another and external services. Moreover, you can also visualize factors such as argument variables, return values, and errors. Thundra's Java agent also allows you to monitor line-by-line tracing where you can actually visualize the behavior of parameters at each line of code.

You can provide instrumentation by either manual instrumentation or automated instrumentation. Thundra’s Java agent integrates with the OpenTracing API and by manual instrumentation, you can manually configure the functions and areas of the function you would like to trace. Automated instrumentation, on the other hand, is simpler to define in the sense that you do not have to specify each function that you need to be traced. The Thundra agent does it for you with minimum configurations required. Moreover, you can also change automated instrumentation configuration with Lambda environment variable without requiring a new deployment.

Manual Instrumentation with Open Tracing

Manual instrumentation required you to define the functions and parts of your lambda functions that you would like to monitor. This also allows you to define custom spans more relevant to the issues you would like to monitor. Furthermore, as Thundra's Java agent complies with the OpenTracing API, you can also use familiar OpenTracing syntax in defining your traces.

By requesting tracing support, you can specify which fields of the request should be traced or how the request should be traced. For example, if you want to trace only id field of the UserGetRequest, you can customize request masking behavior of trace support by overriding the trace plugin as shown in the following example:

...

import io.thundra.agent.lambda.core.handler.request.LambdaRequestHandler;
import io.thundra.agent.lambda.core.handler.LambdaHandlerConfig;
import io.thundra.agent.lambda.core.handler.plugin.LambdaHandlerPlugin;

import io.thundra.agent.lambda.trace.handler.plugin.TraceLambdaHandlerPlugin;

...

public class UserGetHandler 
        implements LambdaRequestHandler<UserGetRequest, UserGetResponse> {

    ...

    @Override
    public LambdaHandlerConfig<UserGetRequest, UserGetResponse> getConfig() {
        return new LambdaHandlerConfig<UserGetRequest, UserGetResponse>() {
            @Override
            public List<LambdaHandlerPlugin<UserGetRequest, UserGetResponse>> getPlugins() {
                return Arrays.asList(
                        new TraceLambdaHandlerPlugin<UserGetRequest, UserGetResponse>() {
                            
                            ...

                            @Override
                            protected Object maskResponse(UserGetResponse response) {
                                return new HashMap<String, Object>() {{
                                    put("id", response.getUser() != null ? response.getUser().getId() : null);
                                }};
                            }
                            
                            ...

                        });
            }
        };
    }

    ...

}

Automated Instrumentation

For automated instrumentation based tracing support, in addition to thundra-agent-trace-instrument, thundra-agent-trace-instrument module must be added as a dependency:

<dependencies>
    ...

    <dependency> 
        <groupId>io.thundra.agent</groupId> 
        <artifactId>thundra-agent-trace-instrument</artifactId> 
        <version>${thundra.version}</version>
    </dependency> 

    ...
<dependencies>
dependencies { 
    ...

    compile group: 'com.opsgenie.thundra, 
            name: 'thundra-audit-instrument', 
            version: '${thundra.version}' 

    ...
}

There are two ways to configure trace support:

  • By annotation: Classes and methods to be traced can be specified and configured by @io.thundra.agent.trace.instrument.config.Traceable annotation.
  • By environment variable: Classes and methods to be traced can be specified and configured by thundra_agent_trace_instrument_traceableConfig environment variable.

Moreover, in general, automated instrumentation allows trace capabilities only on public classes and methods, but private entities can also be traced using the 'method modifier' configurations.

Note: Thundra doesn’t support tracing Lambda handler itself by instrumentation (it is already traced by wrapping the invocation). So don’t use it in the manner seen in the code snippet below:

...

import io.thundra.agent.trace.instrument.config.Traceable;

...

@Traceable
public class UserGetHandler 
        ... {


    ...
      
}

Configuration via Annotations

Classes and methods can be specified by @Traceable annotation to be traced. Classes to be traced must be marked with this annotation. Additionally, methods can be marked as well. Notice that class level @Traceable annotation is mandatory. Otherwise, the class is not checked whether it is trace.

  • If the annotation is used on the class, all public methods of the class are traced by default.
...

import io.thundra.agent.trace.instrument.config.Traceable;

...

@Traceable
public class UserService {

    ...

}

For the UserService, get, save and delete methods are traced (because they are public), but for example getFromCache and getFromRepository are not (because it is private). Additionally, you can specify modifiers of methods to be traced as explained in the next sections.

  • If the annotation is used on the method, the specified method is also traced.
...

import io.thundra.agent.trace.instrument.config.Traceable;

...

@Traceable
public class UserService {

    ...

    @Traceable
    private User getFromCache(String id) {
        ...
    }

    @Traceable
    private User getFromRepository(String id) {
        ...
    }

    ...
}

In this example, in addition to get, save, delete methods, getFromCache and getFromRepository methods are also traced.

  • If you want to specify the method to be traced explicitly, configure class level @Traceable annotation as just marker by setting justMarker property and annotated the desired method with method level @Traceable annotation.
...

import io.thundra.agent.trace.instrument.config.Traceable;

...

@Traceable(justMarker=true)
public class UserService {

    @Traceable
    public User get(String id) {
        ...
    }

    @Traceable
    public User save(User user) {
        ...
    }

    @Traceable
    public User delete(String id) {
        ...
    }

    ...

}

Configuration via Environment Variables

Environment variable name must be in the thundra_agent_trace_instrument_traceableConfig* format. It means that name must start with thundra_agent_trace_instrument_traceableConfig. For example, thundra_agent_trace_instrument_traceableConfig, thundra_agent_trace_instrument_traceableConfig1, thundra_agent_trace_instrument_traceableConfigx, etc …

Environment variable value must be in the <class-def>.<method-def>[propName1=propValue1,propName2=propValue2,...] format where the property definitions are optional. Asterisk character (*) in the <class-def> and <method-def> is supported. For example:

  • get method in the com.mycompany.UserService class: com.mycompany.UserService.get
  • All methods starts with validate in the class com.mycompany.UserService: io.thundra.lambda.demo.service.UserService.validate*
  • All methods in the io.thundra.lambda.demo.service.UserService class: io.thundra.lambda.demo.service.UserService.*
  • All methods in io.thundra.lambda.demo.service package: io.thundra.lambda.demo.service.*.*

Order of `thundra_agent_trace_instrument_traceableConfig`

thundra_agent_trace_instrument_traceableConfig environment variables can be ordered. If a definion’s order is lower than another one, it is checked before. So a thundra_agent_trace_instrument_traceableConfig definition can override another one which has higher order. Default order is 0. There are two ways to configure order of an thundra_agent_trace_instrument_traceableConfig environment variable:

  • By order property. For example io.thundra.lambda.demo.service.UserService.*[...,...,order=1] means that order of the definition is 1
  • By number post-fix after thundra_agent_trace_instrument_traceableConfig environment variable name. For example thundra_agent_trace_instrument_traceableConfig1 means that order of the definition is 1.

Additionally, by thundra_agent_trace_instrument_traceablePrefixes environment variable, you can narrow down the scope of classes to be checked for reducing instrumentation overhead (reduces cold-start overhead as well). Multiple prefixes can be specified by splitting with a comma (,).

This configuration is strongly recommended for starting up Lambda applications faster. For example, you can set thundra_agent_trace_instrument_traceablePrefixes environment variable to io.thundra.lambda.demo for checking only classes from io.thundra.lambda.demo package.

Let’s assume that we have configured UserService for tracing only public methods. In this case, for the get operation, only get method in the UserService is traced.

Method Modifiers

Then we wanted to trace private methods also. In this case, method modifiers can be specified to allow private methods to be traced also.

Configuration via Annotations

methodModifiers attribute of @Traceable annotation can be configured to specify modifiers of methods to be traced. In here modifiers are the flags defined in java.lang.reflect.Modifier class. ANY_METHOD_MODIFIER constant in the io.thundra.agent.trace.instrument.config.TraceableConfig can be used to trace any defined method in a class.

For example, if we want to trace all public and private methods in the UserService class.

...

import io.thundra.agent.trace.instrument.config.Traceable;
import java.lang.reflect.Modifier;

...

@Traceable( 
    methodModifiers = { Modifier.PUBLIC, Modifier.PRIVATE } 
)
public class UserService {

    ...

}

By Environment Variable

To configure method modifiers through environment variable, methodModifiers property of thundra_agent_trace_instrument_traceableConfig environment variable can be configured as hexadecimal format definition of modifiers from java.lang.reflect.Modifier class as shown below:

  • 0x00000001 means public methods
  • 0x00000002 means private methods
  • 0x00000004 means protected methods
  • 0x00000008 means static methods
  • 0x00000001 | 0x00000002 means private or public methods. As shown here, multiple modifiers can be combined by | character.
  • 0x70000000 means any method

Note that as shown in the examples above, multiple modifiers can be combined with the | character and it means logical OR.

In our example, if we want to trace all public and private methods in the UserService class over environment variables, thundra_agent_trace_instrument_traceableConfig environment variable can be specified as io.thundra.lambda.demo.service.UserService.*[...,...,methodModifiers=0x00000001|0x00000002].

After configuring Thundra to trace both private and public methods of a class such as UserService class, you can see more detailed and deeper traces.