Improving Code Design With OpenTelemetry — A Practical Guide | by Roni Dover | May, 2022

What each developer must learn about observability and the right way to leverage OSS instruments to make your code higher

That is the second in a three-part sequence that may evaluation a few of the observability applied sciences accessible to builders in the present day, why they make coding higher, and what particular insights they will present. Right here is the hyperlink to part one in case you missed it.

Within the earlier submit, we mentioned the function of observability within the trendy developer stack. How measuring and learning runtime conduct might help validate our code assumptions — in an analogous technique to checks.

This time, we’ll go away the speculation apart and give attention to the right way to make that occur. I’ll be utilizing a pattern software for reference, however nothing that will likely be mentioned right here is application- or language-specific. I’m additionally eager to study your personal interpretations and purposes to different methods and toolings.

With the intention to reveal extra reasonable eventualities, I sought an software that goes past a easy CRUD implementation and primary scaffolding. That quest proved rather more tough than I anticipated. It seems that it’s not that trivial to seek out good samples with actual area logic depth and even purposes that mix a extra full stack of instruments and platforms.

Ultimately — and maybe inevitably — I used to be drawn to creating my very own pattern primarily based on a template I discovered on-line. You’ll find the unique template repository here. I selected a primary setup that depends on open supply platforms and libraries:

I as soon as made a ‘cash switch’ instance service I used to be keen on, largely due to the flexibility so as to add some logic, validation, and extra exterior processes that made it attention-grabbing. For this submit, I made a decision so as to add some extra characters to my authentic pattern. To make the train much less bland, we’ll be engaged on an API for the Gringotts Wizarding Bank!


The theme gives loads of alternatives so as to add complexity and obstacles that may put some extra meat on our code. Two fast disclaimers: One, I’m not an knowledgeable on HP lore, so bear with me for improvising. And two, this software isn’t imagined to be a mannequin of the right way to construction a well-architected app. Fairly the opposite, we need to see how dangerous design choices will mirror in our observations.


git clone
  • Have Python 3.8+ put in
  • Ensure you have Docker and Docker Compose put in. We’ll use each of them to quick observe by means of the setup and configuration.
  • Use VS Code if doable, as a few of the later examples will depend on that.

Turning code observability ON in two fast steps

  1. We’ll need to launch the observability instruments we’ll use in our instance. Utilizing a little bit of docker-compose, this step is trivial. We’ll be spinning up a number of containers:
  • A Jaeger occasion. We’ll use Jaeger to visualise our distributed traces. We’ll be launching an all-in-one model of Jaeger that’s suited to run as an area occasion.
  • An OpenTelemetry collector. You’ll be able to take into consideration this part extra merely as an observability router. Utilizing a collector is optionally available however gives the advantage of having the ability to modify our observability supply, targets, and charges with out making any code adjustments. It has a separate configuration file, which defines the best way the collector will obtain traces (from our software) and export them to the jaeger occasion.
  • Digma for steady suggestions — We’ll focus on Digma at better size in direction of the top of this submit.

To launch all the pieces, merely run the next instructions from the repo root folder:

docker compose -f ./observability/tracing/docker-compose.hint.yml up -d
docker compose -f ./observability/digma-cf/docker-compose.digma.yml up -d

As soon as all the pieces is up and operating, go to http://localhost:16686/ to test the Jaeger occasion is up. Right here’s what you’ll see:

That’s it. No information but, however the tracing backend is prepared!

2. Now that our hint backend is up and operating, it’s time to instrument our software to ship information to it. That is the place we reap the advantages of OpenTelemetry’s widespread adoption. It’s merely a matter of specifying the fitting packages and activating the instrumentation within the code. The instance here’s a Python stack, however you’ll discover comparable packages for many different languages from GoLang to .NET, Java, or Javascript.

In our case, I’ve added the next packages to the venture necessities file. It’s a handful:

  • opentelemetry-instrumentation-fastapi
  • opentelemetry-instrumentation-logging
  • opentelemetry-instrumentation-asyncpg
  • opentelemetry-instrumentation-pika
  • opentelemetry-instrumentation-pika
  • opentelemetry-instrumentation-httpx
computerized instrumentation of code

The diagram above exhibits the breadth of protection for the automated instrumentation accessible with frequent platforms and libraries. Every purple rhombus represents a tiny probe that’s already instrumented for OpenTelemetry and is able to begin transmitting. With a lot information at hand, it turns into much less of a matter of acquiring data on runtime utilization and extra of the right way to put it to make use of to get to the fitting conclusions.

Configuring OTEL

Turning all of that instrumentation on is easy. First, we add some primary OpenTelemetry setup that consists of specifying some primary data on what we’re tracing and the way we need to export the information. We‘ll be utilizing the usual vanilla implementation of the totally different elements that comes with the OTEL package deal. The code beneath configures OTEL to ship out all the observability information to our ‘router,’ the collector container we began beforehand listening to at localhost:4317.

Moreover, you’ll be able to see some calls to totally different instrument() features, which mainly activate every of the automated instrumentation packages we included in our venture. All in all, fairly normal boilerplate code.

As I discussed within the earlier submit, it’s not within the scope of this submit to go deeper into the setup and configuration of OpenTelemetry. The OTEL web site has great documentation on the subject.

With nice instrumentation comes nice observability. #showerthoughts Tweet by Michael Hausenblas

Now we get began! Our pattern app is a straightforward API service with some added logic to make issues attention-grabbing. The API gives a contemporary manner for wizards to entry their vault, test their ‘steadiness’, and even order an appraisal of its content material.

Let’s set up the appliance necessities (it is really helpful to make use of a digital Python atmosphere. On this instance, we’ll use venv):

python -m venv ./venv
supply ./venv/bin/activate
pip set up -r ./gringotts/necessities.txt

Open the appliance within the IDE, and don’t overlook to change the interpreter to make use of the venv atmosphere we created.

Begin the appliance for the IDE or command line, or simply use docker-compose once more to get it operating utilizing this code:

docker compose --profile standalone -f docker-compose.yml -f docker-compose.override.standalone.yml up -d

Run the next to seed the appliance with some information we will play with. We are able to run ./seed/ straight or simply launch it from a container, as proven beneath:

docker compose -f ./docker-compose.seed.yml up --attach gt-seed-data

The script will import and generate some information, which can also be primarily based on a Harry Potter dataset I discovered online.

We now have a working API to play with at HTTP://localhost:8238/docs.

It is already there. A lot of it’s offered by the automated instrumentation we reviewed earlier than, and we’ve already added some tracing within the code. OpenTelemetry permits us to outline Spans. Spans characterize a logical breakdown of the general means of dealing with a request into significant, granular items.

For instance, when authenticating a buyer at Gringotts, the method may embrace checking their vault key first, authenticating their id, after which validating that the vault quantity the shopper requested to entry certainly belongs to them in accordance with the information. Every of those steps will be represented as a separate span, and it’s significant to grasp and observe its conduct and efficiency.

That is what the Span declaration seems to be like in our code; the start_as_current_span operate declares a logical unit referred to as ‘Authenticate vault proprietor and its key, which we’ll now be capable to observe. In a really comparable technique to writing log messages, we will regularly add increasingly more tracing into the code and thereby enhance our capability to trace its interior workings.

Let’s generate some information to see what that appears like. We are able to set off a number of API operations like logging in by way of the swagger ‘authenticate’ button (username: hpotter, password: griffindoor).

Alternatively, we will run some checks that may already generate loads of information to have a look at. We are able to run our checks utilizing the Pytest command line or simply launch the take a look at suite by way of docker-compose. Discover that we’re additionally seeding information earlier than operating the checks to create extra reasonable circumstances and hopefully higher information. Right here’s the code:

PYTEST_ARGUMENTS="--seed-data true" docker compose --profile take a look at -f docker-compose.yml -f docker-compose.override.take a look at.yml up --attach gt-vault-api --abort-on-container-exit

Now, let’s take a look at what our observability seems to be like. Open your Jaeger occasion at http://localhost:16686. Within the Jaeger UI, we will choose the ‘vault_service’ service and the “/gringotts/vaults/token” or “/gringotts/vaults/authentication” operations.

If we develop the span row that appears to be taking probably the most time, we’ll discover an apparent downside there, which you’ll see beneath:

Appears like repeated SQL calls attributable to suboptimal implementation. If we take a look at the code, it’s instantly obvious somebody applied this particular part of the code within the worst manner doable. Maybe job safety?

Goblin employee processes. Supply:

If we filter the Jaeger interface to have a look at the ‘Appraise’ operation, we’ll be capable to see how distributed tracing truly connects the dots between the totally different software elements. We are able to look at the entire image of the request lifecycle. complicated methods with a number of microservices at work with asynchronous in addition to synchronous flows. Under, we will see the handover between the FastAPI and the ‘GoblinWorker’ service by way of the RabbitMQ queue.

Multiservice movement monitoring

With this information in hand, it’s doable to begin measuring and checking code adjustments and validating what we predict mounted the difficulty. Not solely that, as we’ll focus on within the subsequent submit within the sequence, we will evaluate these traces to precise CI/staging/manufacturing information to establish traits and measure whether or not our repair truly labored underneath real-life circumstances and information.

That is the primary downside with accessing tracing as a developer. There’s a wealth of knowledge, however it’s laborious to know when to discover it and the right way to get to the fitting conclusions. The extra attention-grabbing insights truly come to gentle not by analyzing a single hint however when aggregating and evaluating a number of comparable traces that in some way behave otherwise. This helps us perceive why some customers are seeing extra errors or are experiencing poor efficiency.

That is the place steady suggestions matches in. Particularly, the flexibility to repeatedly analyze these traces utilizing bottom-line sort conclusions, simply as we’d seek the advice of our CI construct. If we may mechanically be alerted of a number of queries being referred to as inefficiently, rising slower with time, or that the dimensions issue of all the request is deteriorating (efficiency per name), that may permit us to optimize our code higher.

The final instrument I needed to debate and reveal in the present day can also be the one I care about probably the most. Full disclosure: I’m the creator of Digma. I’ve written concerning the want for it as early as September of final 12 months. I really feel snug showcasing it right here as a result of it’s each open supply/free and never but launched formally. Additionally, it actually encapsulates my very own ideas about what steady suggestions may turn out to be.

I’ll additionally add a disclaimer to the disclosure: that is an early pre-beta launch, so don’t run it in manufacturing simply but!

To see steady suggestions in motion, we will set up the VS Code plugin from the marketplace. In the event you recall, we already deployed its backend beforehand as part of the tracing stack.

If the OpenTelemetry packages and libraries we enabled mechanically instrumented our code, Digma mechanically ingests that information to provide insights. After opening the workspace, we will now toggle observability information for any operate or space of the code. For instance, here’s what it suggests concerning the ‘authenticate’ operate we mentioned earlier than:

We are able to ignore a few of the extra production-oriented insights for now (degree of visitors, for instance), however even these easy items of knowledge make it simpler to provide, evaluation, and launch the code. As a substitute of evaluating traces or skulking round totally different dashboards, we now have the code insights accessible right here as a result of they’re related to the code we’re working with.

We’ll see what occurs within the remaining submit within the sequence once we throw further information into the combination — manufacturing, staging, and CI. We are able to derive much more related and goal insights to test our code with these observability sources. We’ll be capable to establish phenomena distinctive to manufacturing and measure and enhance characteristic maturity ranges.

You'll be able to attain me on Twitter at @doppleware or right here.
Comply with my open-source venture for steady suggestions at

More Posts