Thundra

Thundra: Serverless Observability for AWS Lambda

The black box nature of AWS Lambda and other serverless environments means that identifying and fixing performance issues is difficult and time-consuming. Built for straightforward debugging, monitoring, and observability, Thundra provides deep insight into your entire serverless environment. Thundra collects and correlates all your metrics, logs, and traces, allowing you to quickly identify problematic invocations and also analyzes external services associated with that function. With Thundra’s zero overhead and automated instrumentation capabilities, your developers are free to write code without worrying about bulking up their Lambdas or wasting time on chasing black box problems.

Get Started    Discussions

Trace Support

Using traces, you can gain end-to-end visibility on what's happening inside your functions. It helps you to pinpoint the problematic areas and latencies in your application. You can find the root cause of an error by debugging your function line by line when needed.

Trace support in the Java agent can be used by both Automated and Manual instrumentation, as mentioned in instrumentation.

Using Trace with Automated Instrumentation

Automated instrumentation allows you to gain trace Support without specifying each method and or class in detail to monitor as is the issue with manual instrumentation. After configuring your Lambda functions with automated instrumentation as shown in the Instrumentation guides, you can further configure your traces as are mentioned below:

By adding thundra-agent-lambda-trace module to your project, Thundra can trace Lambda invocations.

<dependencies>
    ...

    <dependency> 
        <groupId>io.thundra.agent</groupId> 
        <artifactId>thundra-agent-lambda-trace</artifactId> 
        <version>${thundra.agent.version}</version>
    </dependency> 

    ...
<dependencies>
dependencies { 
    ...

    compile group: 'com.opsgenie.thundra, 
            name: 'thundra-lambda-audit', 
            version: '${thundra.version}' 

    ...
}

By default, request and response of the invocations are traced. You can see the following invocation detail page with the basic set up of tracing.

Let's look at the possible ways of enriching traces

Disable request tracing

By setting thundra_agent_lambda_trace_request_skip environment variable to true (by default it is false), tracing invocation request can be skipped, so request data will not be placed in the trace data.

Disable Response Tracing

By setting thundra_agent_lambda_trace_response_skip environment variable to true (by default it is false), tracing invocation response can be skipped, so response data will not be placed in the trace data.

Configure Trace Level

You can define levels for classes/methods to be trace. If the defined level of the class or method is not less than the global trace level, it is traced which otherwise is skipped. The motivation behind this feature is the same as the log levels. By default, low-level methods are not traced, but in the case of detailed tracing, these methods are also taken into consideration while tracing for collecting more granular information. Global trace level is 0 by default and can be configured through thundra_agent_trace_instrument_traceLevel environment variable.

In this example, the followings are configured,

  • validation methods with lowest level 0
  • cache methods with higher level 1
  • repository methods with level 2
  • all the other methods (class level configuration is applied to all methods but it is overridden by method level configuration) with highest level 3

By Annotation

traceLevel attribute of @Traceable annotation can be set to any integer to configure trace level of the associated class or method.

...

import io.thundra.agent.trace.instrument.config.Traceable;

...

@Traceable(
    ...
    traceLevel = 3 
)
public class UserService {

    ...

    //////////////////////////////////////////////////////////////////////

    @Traceable(
        ...
        traceLevel = 0 
    )
    private void validateId(String id) {
        ...
    }

    @Traceable(
        ...
        traceLevel = 0 
    )
    private void validateUser(User user) {
        ...
    }

    //////////////////////////////////////////////////////////////////////

    @Traceable(
        ...
        traceLevel = 1 
    )
    private User getFromCache(String id) {
        ...
    }

    @Traceable(
        ...
        traceLevel = 1 
    )
    private void putToCache(User user) {
        ...
    }

    @Traceable(
        ...
        traceLevel = 1 
    )
    private void deleteFromCache(String id) {
        ...
    }

    //////////////////////////////////////////////////////////////////////

    @Traceable(
        ...
        traceLevel = 2 
    )
    private User getFromRepository(String id) {
        ...
    }

    @Traceable(
        ...
        level = 2 
    )
    private void putToRepository(User user) {
        ...
    }

    @Traceable(
        ...
        traceLevel = 2 
    )
    private User deleteFromRepository(String id) {
        ...
    }

    //////////////////////////////////////////////////////////////////////

    public User get(String id) {
        ...
    }

    public void save(User user) {
        ...
    }


    public User delete(String id) {
        ...
    }

}

By Environment Variable

To configure trace level through environment variable, traceLevel property of thundra_agent_trace_instrument_traceableConfig environment variable can be set to any integer.

In this example, if you want to set trace level of methods in the UserService class over environment variables, thundra_agent_trace_instrument_traceableConfig environment variable can be specified as io.thundra.lambda.demo.service.UserService.*[...,...,traceLevel=3].

Additionally, for the example above, the trace level can be configured per method basis through environment variables as following:

  • thundra_agent_trace_instrument_traceableConfig1: io.thundra.lambda.demo.service.UserService.*[traceLevel=3]
  • thundra_agent_trace_instrument_traceableConfig2: io.thundra.lambda.demo.service.UserService.validate*[traceLevel=0]
  • thundra_agent_trace_instrument_traceableConfig3: io.thundra.lambda.demo.service.UserService.*Cache[traceLevel=1]
  • thundra_agent_trace_instrument_traceableConfig4: io.thundra.lambda.demo.service.UserService.*Repository[traceLevel=2]

By default global trace level is 0 so all methods are trace. But if you set global trace level to 3 only get, save and delete methods are trace. But, if the global trace level is set to 2 (set thundra_agent_trace_instrument_traceLevel to 2) and in this case, in addition to get, save and delete methods, repository logic related methods getFromRepository, putToRepository, deleteFromRepository methods will also be traced.

Using Trace with Manual Instrumentation

Custom Tag Injection

In addition to automated trace details, Thundra also supports user-provided custom tags in key/value format. You can inject your custom tags into the spans.

You can access the current span through the global ThundraTracer retrieved from io.thundra.agent.trace.TraceSupport and then inject your custom tag. In this example, in the get operation, we inject the retrieved user as get.user named tag into the span of get method.

...

import io.thundra.agent.trace.instrument.config.Traceable;
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.ThundraTracer;
import io.thundra.agent.trace.span.ThundraSpan;

@Traceable(
    ...
)
public class UserService {

    private final ThundraTracer tracer = TraceSupport.getTracer();
  
    ...

    public User get(String id) {
        ...

        ThundraSpan span = tracer.activeSpan();   
        span.setTag("get.user", user);

        return user;
    }

    ...

}

After you inject your custom tags, you can see them under the tags of the span (which is get method in our example).

Custom Spans

Automated instrumentation is good but not enough. So we added custom instrumentation support to give you flexibility for defining your sub-contexts.

Through the global ThundraTracer retrieved from io.thundra.agent.trace.TraceSupport, we can create our custom span. In this example, in the get operation, we create our custom span to instrument stat update behaviour as the sub-span of the get method’s span.


...

import io.thundra.agent.trace.instrument.config.Traceable;
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.ThundraTracer;
import io.thundra.agent.trace.span.ThundraSpan;

@Traceable(
    ...
)
public class UserService {

    private final ThundraTracer tracer = TraceSupport.getTracer();
    
    ...

    public User get(String id) {
        ...

        ThundraSpan span = tracer.buildSpan("update-stats").startActive(true).span();
        try {
            if (user != null) {
                span.setTag("hit", true);
                // increase hit stats
                ...
            } else {
                span.setTag("hit", false);
                // increase miss stats
                ...
            }
        } finally {
            span.finish();
        }

        return user;
    }

    ...

}

After you manually instrument updating stats behaviour in the get method, you can see the instrumented span under the span of get method.

Customize request tracing (Masking)

By request tracing customization support, you can specify which fields of the request should be traced or how the request should be traced.

For example, if you want to trace only id field of the UserGetRequest, you can customize request masking behavior of trace support by overriding the trace plugin as shown in the following example:

....

import com.amazonaws.services.lambda.runtime.Context;

import io.thundra.agent.lambda.core.handler.request.LambdaRequestHandler;
import io.thundra.agent.lambda.core.handler.LambdaHandlerConfig;
import io.thundra.agent.lambda.core.handler.plugin.LambdaHandlerPlugin;

import io.thundra.agent.lambda.trace.handler.plugin.TraceLambdaHandlerPlugin;

...

public class UserGetHandler 
        implements LambdaRequestHandler<UserGetRequest, UserGetResponse> {

    ...

    @Override
    public LambdaHandlerConfig<UserGetRequest, UserGetResponse> getConfig() {
        return new LambdaHandlerConfig<UserGetRequest, UserGetResponse>() {
            @Override
            public List<LambdaHandlerPlugin<UserGetRequest, UserGetResponse>> getPlugins() {
                return Arrays.asList(
                        new TraceLambdaHandlerPlugin<UserGetRequest, UserGetResponse>() {
                            
                            ...

                            @Override
                            protected Object maskRequest(UserGetRequest request) {
                                return new HashMap<String, Object>() {{
                                   put("id", request.getId());
                                }};
                            }

                            ...
                            
                        });
            }
        };
    }

    ...

}

Note: Same way is also supported over LambdaRequestStreamHandler.

Customize response tracing

By response tracing customization support, you can specify which fields of the response should be traced or how the response should be traced.

For example, if you want to trace only id field of the User in the UserGetResponse, you can customize response masking behavior of trace support by overriding trace plugin as shown in the following example:

...

import io.thundra.agent.lambda.core.handler.request.LambdaRequestHandler;
import io.thundra.agent.lambda.core.handler.LambdaHandlerConfig;
import io.thundra.agent.lambda.core.handler.plugin.LambdaHandlerPlugin;

import io.thundra.agent.lambda.trace.handler.plugin.TraceLambdaHandlerPlugin;

...

public class UserGetHandler 
        implements LambdaRequestHandler<UserGetRequest, UserGetResponse> {

    ...

    @Override
    public LambdaHandlerConfig<UserGetRequest, UserGetResponse> getConfig() {
        return new LambdaHandlerConfig<UserGetRequest, UserGetResponse>() {
            @Override
            public List<LambdaHandlerPlugin<UserGetRequest, UserGetResponse>> getPlugins() {
                return Arrays.asList(
                        new TraceLambdaHandlerPlugin<UserGetRequest, UserGetResponse>() {
                            
                            ...

                            @Override
                            protected Object maskResponse(UserGetResponse response) {
                                return new HashMap<String, Object>() {{
                                    put("id", response.getUser() != null ? response.getUser().getId() : null);
                                }};
                            }
                            
                            ...

                        });
            }
        };
    }

    ...

}

Note: Same way is also supported over LambdaRequestStreamHandler.

Additional Configurations

The trace support provided by Thundra's Java agent can provide a more in-depth monitoring experience by the configuration of the features below:

  • Trace Arguments
  • Return Values
  • Trace Errors
  • Line-by-Line Tracing
  • Integration with Other Libraries: Thundra Java agent allows you to instrument your functions to trace for many popular frameworks and libraries by using a integrations system. By default, all built-in integrations are disabled. Integrations can be enabled with a simple configuration which involves adding dependencies in either your Maven or Gradle. Here is the list of integration names and supported versions :
Integration Name
Library Link
Supported Version

aws.amazon.com/sdk-for-java/

docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java.html

hc.apache.org

  • PostgreSQL `: jdbc.postgresql.org
  • MySQL: dev.mysql.com
  • MariaDB: mariadb.com/kb/en/library/about-mariadb-connector-j/
  • Microsoft SQL Server: docs.microsoft.com/en-us/sql/connect/jdbc/microsoft-jdbc-driver-for-sql-server?view=sql-server-2017
  • PostgreSQL :
  • MySQL :
  • MariaDB :
  • Microsoft SQL Server :
  • Jedis: javadoc.io/doc/redis.clients/jedis/2.9.0
  • Lettuce: lettuce.io
  • Redisson: redisson.org
  • Jedis:
  • Lettuce:
  • Redisson: