Enrich Tracing

Manual Instrumentation with Open Tracing

Automated instrumentation is good but not always enough. Sometimes you may want to instrument on your own. Through the global ThundraTracer retrieved from io.thundra.agent.trace.TraceSupport, you can create your custom spans. In this example, we create our custom span to instrument getting user operation.

UserService
...
import io.thundra.agent.trace.TraceSupport;
import io.thundra.agent.trace.ThundraTracer;
import io.thundra.agent.trace.span.ThundraSpan;
...
public class UserService {
private final ThundraTracer tracer = TraceSupport.getTracer();
...
public User get(String id) {
ThundraSpan span = tracer.buildSpan("user.get").startActive(true).span();
try {
if (user != null) {
span.setTag("hit", true);
span.setTag("get.user", user);
} else {
span.setTag("hit", false);
...
}
} finally {
span.finish();
}
return user;
}
...
}

Automated Instrumentation

There are two ways to configure audit (trace) support:

  • By Annotation: Classes and methods to be audited can be specified and configured by @io.thundra.agent.trace.instrument.config.Traceable

    annotation.

  • By Environment variable: Classes and methods to be trace can be specified and configured by thundra_agent_trace_instrument_traceableConfig environment variable.

Note: Thundra doesn’t support tracing Lambda handler itself by instrumentation (it is already traced by wrapping the invocation). So don’t use it in the manner seen in the code snippet below:

Specify classes/methods to be traced

By Annotation

Classes and methods can be specified by @Traceable annotation to be traced. Classes to be traced must be marked with this annotation. Additionally, methods can be marked as well. Notice that class level @Traceable annotation is mandatory. Otherwise, class is not checked whether it is traced.

  • If the annotation is used on the class, all public methods of the class are traced by default.

UserService.java
...
import io.thundra.agent.trace.instrument.config.Traceable;
...
@Traceable
public class UserService {
...
public User get(String id) {
...
}
}

In this example, public get method is traced automatically.

  • If the annotation is used on the method, the specified method is also traced.

UserService.java
...
import io.thundra.agent.trace.instrument.config.Traceable;
...
@Traceable
public class UserService {
...
@Traceable
private User getFromCache(String id) {
...
}
@Traceable
private User getFromRepository(String id) {
...
}
...
}

In this example, in addition to getmethod, getFromCache and getFromRepository methods are also traced.

If you want to specify method to be traced explicitly, configure class level @Traceable annotation as just marker by setting justMarker property and annotated the desired method with method level @Traceable annotation.

UserService.java
...
import io.thundra.agent.trace.instrument.config.Traceable;
...
@Traceable(justMarker=true)
public class UserService {
@Traceable
public User get(String id) {
...
}
...
}

In this example, only get method is traced. Because class level @Traceable annotation is marker and only getmethod is annotated.

By Environment Variable

Environment variable name must be in the thundra_agent_trace_instrument_traceableConfig* format. It means that name must start with thundra_agent_trace_instrument_traceableConfig. For example, thundra_agent_trace_instrument_traceableConfig, thundra_agent_trace_instrument_traceableConfig1, thundra_audit_auditableDefx, etc …

Environment variable value must be in the <class-def>.<method-def>[propName1=propValue1,propName2=propValue2,...] format where the property definitions are optional. Asterisk character (*) in the <class-def> and <method-def> is supported. For example:

  • get method in the com.mycompany.UserService class: com.mycompany.UserService.get

  • All methods starts with validate in the class com.mycompany.UserService: com.mycompany.UserService.validate*

  • All methods in the com.mycompany.UserService class: com.mycompany.UserService.*

  • All methods in com.mycompany package: com.mycompany.*.*

thundra_agent_trace_instrument_traceableConfig environment variables can be ordered. If a definition’s order is lower than another one, it is checked before. So a thundra_agent_trace_instrument_traceableConfig definition can override another one which has higher order. Default order is 0. There are two ways to configure order of an thundra_agent_trace_instrument_traceableConfig environment variable:

  • By order property. For example com.mycompany.UserService.*[...,...,order=1] means that order of the definition is 1

  • By number post-fix after thundra_agent_trace_instrument_traceableConfig environment variable name. For example thundra_agent_trace_instrument_traceableConfig1 means that order of the definition is 1.

Additionally, by thundra_agent_trace_instrument_traceablePrefixes environment variable, you can narrow down the scope of classes to be checked for reducing instrumentation overhead (reduces cold-start overhead as well). Multiple prefixes can be specified by splitting with a comma (,).

This configuration is strongly recommended for starting up Lambda applications faster. For example, you can set thundra_agent_trace_instrument_traceablePrefixes environment variable to io.thundra.lambda.demo for checking only classes from io.thundra.lambda.demo package.

Method Modifiers

Then we wanted to trace private methods also. In this case, method modifiers can be specified to allow private methods to be traced also.

Configuration via Annotations

methodModifiers attribute of @Traceable annotation can be configured to specify modifiers of methods to be traced. In here modifiers are the flags defined in java.lang.reflect.Modifier class. ANY_METHOD_MODIFIER constant in the io.thundra.agent.trace.instrument.config.TraceableConfig can be used to trace any defined method in a class.

For example, if we want to trace all public and private methods in the UserService class.

UserService.java
...
import io.thundra.agent.trace.instrument.config.Traceable;
import java.lang.reflect.Modifier;
...
@Traceable(
methodModifiers = { Modifier.PUBLIC, Modifier.PRIVATE }
)
public class UserService {
...
}

By Environment Variable

To configure method modifiers through environment variable, methodModifiers property of thundra_agent_trace_instrument_traceableConfig environment variable can be configured as hexadecimal format definition of modifiers from java.lang.reflect.Modifier class as shown below:

  • 0x00000001 means public methods

  • 0x00000002 means private methods

  • 0x00000004 means protected methods

  • 0x00000008 means static methods

  • 0x00000001 | 0x00000002 means private or public methods. As shown here, multiple modifiers can be combined by | character.

  • 0x70000000 means any method

Note that as shown in the examples above, multiple modifiers can be combined with the | character and it means logical OR.

In our example, if we want to trace all public and private methods in the UserService class over environment variables, thundra_agent_trace_instrument_traceableConfig environment variable can be specified as io.thundra.lambda.demo.service.UserService.*[...,...,methodModifiers=0x00000001|0x00000002]

Trace Arguments can be described as names, types, and values of variables within your Lambda functions. Due to the black box nature of serverless, monitoring these parameters is difficult, yet necessary when debugging and testing your code. Hence one of the aims of Thundra's Trace capabilities is to ensure ease of monitoring of your arguments, and this is done by configuring your Trace as per your needs. Enabling argument support can be done either by annotation or via environment variables.

Configuration via Annotations

traceArguments attribute of the @Traceable annotation can be set to true for tracing method arguments. Additionally, the following properties can be configured for more detailed argument tracing:

  • By default, argument name is generated from argument’s order such as arg-0, arg-1, etc … traceArgumentNames attribute can be set to true for extracting argument names if a local variable table of the associated method is available in the owner class’s bytecode.

  • By default, argument value is generated from the toString method of the argument instance. If you want to serialize the argument value in a structured format such as JSON, you can set serializeArgumentsAsJson attribute to true

...
import io.thundra.agent.trace.instrument.config.Traceable;
...
@Traceable(
...,
traceArguments = true,
tracetArgumentNames = true,
serializeArgumentsAsJson = true
)
public class UserService {
...
}

Configuration via Environment Variables

To configure tracing argument behaviour through environment variable, traceArguments property of thundra_agent_trace_instrument_traceableConfig environment variable can be set to true.

Similarly:

  • Tracing argument names behaviour can be configured by setting traceArgumentNames property of thundra_agent_trace_instrument_traceableConfig environment variable to true

  • Serializing arguments’ values as JSON can be configured by setting serializeArgumentsAsJson to true

If you want to trace arguments (names, types and values) of monitored methods in for example, the UserService class over environment variables, thundra_agent_trace_instrument_traceableConfig environment variable can be specified as io.thundra.lambda.demo.service.UserService.*[...,...,traceArguments=true,traceArgumentNames=true,serializeArgumentsAsJson=true]

Trace Return Values

Overall Thundra's Java Trace support allows you to method level tracing. However, that may not always be sufficient as this does not provide information about the return values of your Lambda Functions. Hence Thundra's Java agent allows you to trace these return values upon your function invocations. By default, the feature is disabled but can be configured either by annotations or via environment variables.

Configuration via Annotations

traceReturnValue attribute of @Traceable annotation can be set to true for tracing return values. By default, the return value is generated from the toString method of the return value instance. If you want to serialize the return value in a structured format such as JSON, you can set serializeReturnValueAsJson attribute to true.

UserService.java
...
import io.thundra.agent.trace.instrument.config.Traceable;
...
@Traceable(
...,
traceReturnValue = true,
serializeReturnValueAsJson = true
)
public class UserService {
...
}

Configuration via Environment Variables

To configure tracing return value behaviour through environment variable, traceReturnValue property of thundra_agent_trace_instrument_traceableConfig environment variable can be set to true. In a similar way, serializing return value as JSON can be configured by setting serializeReturnValueAsJson to true

For example, if you want to trace return values (types and values) of monitored methods in the UserService class over environment variables, thundra_agent_trace_instrument_traceableConfig environment variable can be specified as io.thundra.lambda.demo.service.UserService.*[...,...,traceReturnValue=true,serializeReturnValueAsJson=true]

Trace Errors

Thundra's Trace support allows you to monitor errors cause in each span generated which in turn represents each part of your function being monitored. Monitoring any errors thrown is an integral part of testing and implementing your functions. Moreover, several errors may occur in your Lambda functions that would generally not occur during implementation of the function as the result of edge cases or external services failing. Hence tracing errors is an essential parameter, and it can be monitored using Thundra's Trace support. The feature is disabled by default but can be enabled either using annotations or via environment variables.

Configuration via Annotations

traceError attribute of @Traceable annotation can be set to true for tracing errors thrown from methods.

UserService.java
...
import io.thundra.agent.trace.instrument.config.Traceable;
...
@Traceable(
...,
traceError = true
)
public class UserService {
...
}

Configuration via Environment Variables

To configure tracing error behaviour through environment variable, traceError property of thundra_agent_trace_instrument_traceableConfig environment variable can be set to true.

For example, if you want to trace errors thrown from monitored methods in the UserService class over environment variables, thundra_agent_trace_instrument_traceableConfig environment variable can be specified as io.thundra.lambda.demo.service.UserService.*[...,...,traceError=true].

Line-by-Line Tracing

Line-by-Line tracing is supported by Thundra's Java agent and it allows you to monitor your Lambda functions literally by each line. That means you can see how your functions behave at the important lines of your function giving you in-depth monitoring capabilities. So you can have micro-tracing capability even inside the method. In addition to the duration of each line, with the help of line by line tracing, code execution flow in the method can be traced by which line was executed after which line. By default, line by line tracing behavior is disabled. This feature is disabled by default but can be configured as preferred either using annotations or using environment variables.

Configuration via Annotations

traceLineByLine attribute of @Traceable annotation can be set to true for tracing methods line by line.

UserService.java
...
import io.thundra.agent.trace.instrument.config.Traceable;
...
@Traceable(
...,
traceLineByLine = true
)
public class UserService {
...
}

Configuration via Environment Variables

To configure line by line tracing behaviour through environment variable, traceLineByLine property of thundra_agent_trace_instrument_traceableConfig environment variable can be set to true.

For example, if we want to enable line by line tracing for monitored methods in the UserService class over environment variables, thundra_agent_trace_instrument_traceableConfig environment variable can be specified as io.thundra.lambda.demo.service.UserService.*[...,...,traceLineByLine=true]

Trace Local Variables

Thundra can provide local variable states line by line if a local variable table of the associated method is available in the owner class’s bytecode. Local variable tracing allows you to see the names, types, and values of the alive local variables visible from the current code scope at each line. By this feature, value changes of the local variables can be traced in the method at each line. This is a debugging feature rather than tracing.

Note: that to enable this feature, tracing line by line feature must be enabled as explained above.

Tracing Local Variables Trace

Configuration via Annotations

traceLocalVariables attribute of @Traceable annotation can be set to true for tracing local variables in the method.

UserService.java
...
import io.thundra.agent.trace.instrument.config.Traceable;
...
@Traceable(
...,
traceLineByLine = true,
traceLocalVariables = true
)
public class UserService {
...
}

Configuration via Environment Variables

To configure tracing local variable behaviour through environment variable, traceLocalVariables property of thundra_agent_trace_instrument_traceableConfig environment variable can be set to true.

For example, if we want to trace local variables for monitored methods in the UserService class over environment variables, thundra_agent_trace_instrument_traceableConfig environment variable can be specified as io.thundra.lambda.demo.service.UserService.*[...,traceLineByLine=true,traceLocalVariables=true]

API Gateway Custom Authorizer Support

TOKEN Authorizer

Thundra provides io.thundra.agent.lambda.core.auth.TokenAuthorizerContext class as input type to your custom authorizer. It provides all the required properties like authorizationToken and methodArn. To see your custom authorizer in distributed trace map, you need to use this TokenAuthorizerContext class as input to your custom authorizer handler.

REQUEST Authorizer

Thundra provides io.thundra.agent.lambda.core.auth.RequestAuthorizerContext class as input type to your custom authorizer. It provides all the required properties like methodArn, resource, path, httpMethod, header, queryStringParameters, pathParameters, stageVariables, requestContext, etc ... To see your custom authorizer in distributed trace map, you need to use this RequestAuthorizerContext class as input to your custom authorizer handler...