Trace Context Propagation With OpenTelemetry | by Dmitry Kolomiets | Apr, 2022

Musings on Tracing

Sensible notes on utilizing AWS Distro for OpenTelemetry

Picture by Anne Nygård on Unsplash
Desk of ContentsSummary
Trace context propagation
End to end trace
Dangling AWS Lambda traces

Till this level I used to be speaking about OpenTelemetry and ADOT typically, with out discussing particular programming languages. The next submit is predicated on AWS managed Lambda layer for ADOT Python. I consider that conclusions are usually the identical for different languages.

The purpose for this submit is precisely as outlined in the previous one — display how we will manually go the hint context round when AWS companies don’t assist this natively (for instance, with AWS Kinesis streams). This time we’ll do that with OpenTelemetry, not AWS X-Ray (we’re going to use X-Ray console to assessment the ensuing traces although).

Now when you recognize about OpenTelemetry collectors, distributions, and might inform the distinction between ADOT and ADOT — we will lastly speak about sensible bits.

Evaluation Approaching OpenTelemetry when you want a refresher on these subjects.

To display the hint context propagation with OpenTelemetry I’m going to make use of ADOT Python Lambda layer and OpenTelemetry Python SDK for the next causes:

  • I used Python in the previous post to display hint context propagation with X-Ray SDK. I figured it could be helpful to see how the identical software seems like with OpenTelemery
  • Python is among the languages (Java is one other) that helps automatic instrumentation. It is very important spotlight the variations between computerized and guide instrumentation to know what is going on behind the scenes.

We’ll use the identical software structure we examined earlier than:

The distributed structure we wish to hint with OpenTelemetry

Producer Lambda

We begin with Producer lambda — essentially the most simple one. Let me present you the code first:

Producer lambda implementation (computerized instrumentation)

This suspiciously easy implementation not solely captures the traces for the lambda perform, but additionally devices and captures traces from boto3 library, handles interactions with ADOT Collector, and exports the traces to AWS X-Ray. That is the extent of service that ADOT Lambda layers present when automatic instrumentation is enabled!

Extra particularly, it’s worthwhile to do the next to allow computerized instrumentation:

  • Add ADOT Lambda layer to your perform — this installs OpenTelemetry SDK, ADOT Collector, AWS-specific extensions, and so forth.
  • Enable auto instrumentation by including AWS_LAMBDA_EXEC_WRAPPER variable with /choose/otel-instrument worth

Even with out understanding every thing that occurs behind the scenes, this seems fairly good already.

Automated instrumentation — hint construction

As a sanity test, we will invoke Producer perform within the console (with any payload) and see the ensuing hint in X-Ray console:

Let’s assessment the construction of the captured X-Ray hint — there will probably be an vital distinction after we transfer on to the guide instrumentation later.

X-Ray service makes use of the ideas of segment and subsegment to indicate dad or mum/youngster relationships between models of labor that comprise the hint. OpenTelemetry makes use of an idea of a span for a similar goal — there could also be dad or mum and youngster spans.
Although we use X-Ray console to view the ensuing traces, I’m going to make use of OpenTelemetry phrases on this submit.

The primary two spans are added to the hint by the AWS Lambda service. It is very important perceive that these two spans are captured and exported to X-Ray even earlier than our Producer lambda is invoked.

AWS Lambda exports these two spans instantly to X-Ray. On the time of writing there isn’t a method to configure AWS Lambda and specify one other telemetry backend such Jaeger, Splunk, and so forth.
An implication — if OpenTelemetry is configured with one other telemetry backend, AWS Lambda-specific spans will not be exported there.

The subsequent producer-function span is the span captured by OpenTelemetry — that is the span that’s created by the automated instrumentation startup script (keep in mind that /choose/otel-instrument script we set as AWS_LAMBDA_EXEC_WRAPPER setting variable?).

If we choose producer-function span, we’ll see the kid producing_messages span we manually created in our lambda perform. There are two extra spans for AWS SQS and AWS Kinesis companies as properly — these are captured from boto3 library. When computerized instrumentation is used, the startup script devices the commonest Python libraries to robotically seize spans from them. Be aware that instrumentors ought to be installed and packaged along with your lambda perform to ensure that this to work. You’ll find the comprehensive list of supported libraries on GitHub.

Producer perform hint

Hope you’ve gotten a greater concept of what’s taking place when computerized instrumentation is used. Let’s transfer on to the following lambda.

Client SQS Lambda perform

This perform must extract the hint context from the SQS message and make sure that new spans generated by the perform “proceed” the unique hint for the message. Once more, we’re utilizing computerized instrumentation for this lambda. Right here is the implementation:

As with the Producer lambda, we don’t have to hassle with OpenTelemetry configuration an excessive amount of. We extract traceId and spanId fields from SQS message — this half is equivalent to our previous trace propagation example with X-Ray. There are some things value mentioning although.

Server span form
One other fascinating element is a particular SERVER span form we assign to the consuming_sqs span. That is obligatory to have the ability to see this span as a separate node in X-Ray service map. In line with the documentation:

Be aware that solely spans of form Server are transformed into X-Ray segments, all different spans are transformed into X-Ray subsegments.

Preserve this obscure element in thoughts when you plan to export OpenTelemetry traces to AWS X-Ray. When you use one other telemetry backend — different span kinds might be helpful.

Span hyperlinks
Be aware that we add an optionally available hyperlinks parameter after we create the consuming_sqs span.

The thought is easy — every lambda invocation might contribute to a number of traces and it could be helpful to have a method to correlate them. In our case, we’ve a lambda perform triggered by SQS service. Listed below are the traces concerned when the perform is triggered with a batch of N messages:

  • Lambda invocation hint. When SQS service triggers the lambda perform, an implicit hint is created. That is the “default” hint —hint that you just usually see within the AWS X-Ray console. There will probably be two spans that AWS Lambda service emits by default (we mentioned them above after we examined computerized instrumentation hint construction). This hint may additionally embrace spans for the lambda chilly begin or something that occurs earlier than we begin processing SQS messages
  • SQS message traces at most N of them. Keep in mind that every SQS message might belong to a distinct hint and we wish to “proceed” that unique hint after we course of the message, not create a brand new one

Subsequently, after we course of SQS message (including spans to the unique hint of the message) we wish to hold the hyperlink to the Lambda invocation hint — it could be notably helpful for troubleshooting situations.

Hyperlinks between spans/traces could also be a strong method to categorical causality. It will likely be troublesome to display these hyperlinks in AWS X-Ray console as X-Ray doesn’t assist hyperlinks (but?). Within the subsequent submit, I’ll present how the identical OpenTelemetry hint seems like in a distinct telemetry backend (Jaeger) and can display the usefulness of the hyperlinks then.

Client Kinesis Lambda perform

The final Kinesis shopper lambda is essentially the most tough one primarily because of the issue in AWS managed Lambda layer for ADOT Python that pressured me to desert computerized instrumentation (delete AWS_LAMBDA_EXEC_WRAPPER setting variable). This can be a good factor as I’ll display the way to add OpenTelemetry assist with out ADOT magic. Beware, there are numerous issues to soak up:

Let’s cowl OpenTelemetry initialization step-by-step:

Initialize Tracer Supplier

The very first thing we have to do is to initialize the Tracer Supplier — that is the article that handles the gathering of the traces and export to the OpenTelemetry Collector. Once we initialize the supplier we specify an ID generator compatible with X-Ray (not obligatory if X-Ray assist shouldn’t be wanted) and different attributes we wish to connect to the spans. For this instance, we set an express service title and seize different AWS Lambda-specific attributes (perform ARN, title, reminiscence/CPU allocation, and so forth.)

The Tracer Supplier might be initialized solely as soon as — when you attempt to set one other supplier, the second name will probably be ignored.

Let’s unwrap the following line:

  • add_span_processor —registers a span processor with the Tracer Supplier. Processor is a assemble OpenTelemetry introduces to pre-process knowledge earlier than it’s exported (e.g. modify attributes or pattern) or helps to make sure that knowledge makes it via a pipeline efficiently (e.g. batch/retry). A very good abstract OpenTelemetry processors might be present in Collector GitHub repository.
  • BatchSpanProcessor — the batch processor accepts spans and locations them into batches. Batching helps higher compress the info and scale back the variety of outgoing connections required to transmit the info
  • OTLPSpanExporter — exports spans in OpenTelemetry protocol (OTLP) format. By default, the exporter connects to the native ADOT Collector occasion working alongside your lambda code. It’s the accountability of the ADOT Collector to ahead the spans additional — by default, the collector is configured to go them on to AWS X-Ray service.

To recap — Tracer Supplier manages the gathering and export of the traces to ADOT Collector utilizing BatchSpanProcessor and OTLPSpanExporter. With computerized instrumentation enabled all these low-level particulars are hidden, however now we’ve full management over the processing and exporting of the spans. I’ll use this strategy in future posts to display how we will prolong capabilities supplied by ADOT lambda layers and add assist for added telemetry backends.

Shifting on:

This line allows instrumentation for boto3 API. With guide instrumentation it’s our accountability to instrument the libraries we’re thinking about.

Lastly (the pun supposed), you might have seen this block of code:

That is obligatory to make sure that all spans are correctly exported to the ADOT collector even in case of unhandled exception. That is notably vital when BatchSpanProcessor is used — keep in mind that it batches a number of spans collectively earlier than it forwards them to the collector. With out force_flush name, a number of the spans could also be misplaced (and, after all, these will probably be a very powerful ones)

That is it! Now you need to have an honest psychological mannequin of OpenTelemetry initialization and fundamental configuration and perceive the variations between computerized and guide instrumentation modes supplied by ADOT lambda layer.

Earlier than we wrap up we must always run the Producer lambda perform and test the ensuing end-to-end hint:

In contrast to my previous post, I’m not conscious of any “hacks” or unsupported APIs I used on this instance of hint propagation. This can be a correct implementation based on OpenTelemetry specification. Even higher — we will get end-to-end X-Ray traces with OpenTelemetry.

We briefly touched upon spans exported by AWS Lambda service above. These spans are clearly seen after we have a look at the whole service map in X-Ray:

Client lambdas emit OpenTelemetry spans that “proceed” our major end-to-end hint. On the similar time, we nonetheless have traces emitted by AWS Lambda service for these features. These traces are indifferent from the principle end-to-end hint or “dangling”. They’re seen in X-Ray console due to the direct integration between AWS Lambda and AWS X-Ray companies. Remember that if you’re going to use a distinct telemetry backend with OpenTelemetry, these AWS Lambda spans will not be exported.

We now have coated numerous floor on this submit by displaying the way to combine OpenTelemetry to event-driven AWS architectures and get full end-to-end traces. We’re nonetheless utilizing AWS X-Ray as our major telemetry backend however we do not need to — with OpenTelemetry we’ve a method to export traces to a different backend or occasion to a number of backends on the similar time. That is what we’re going to do within the subsequent submit — I’ll present you the way to combine with Jaeger — a preferred open-source distributed tracing backend.

More Posts