Running Netflix Conductor 3 in Docker Using Elasticsearch 7 and PostgreSQL | by Erik Burger | Mar, 2022

All wrapped in a straightforward, repeatable, and maintainable course of

The usual implementation of Netflix Conductor is configured to make use of Elasticsearch 6. This text covers the adjustments that must be made to make use of Elasticsearch 7 and PostgreSQL, in addition to offering a number of scripts to automate constructing and operating Conductor, each regionally and in a cloud structure.

On the time of writing, the newest launched model of Netflix Conductor is 3.5.3. The data on this article is predicated on model 3.5.1 however ought to be relevant (and simply modified) to the newest model. When you run into any points, please let me know, and I’ll do my finest to assist. One other nice supply of assistance is the Netflix Conductor discussion board.

I’m operating all of the provided code on a Home windows 10 machine, from Git Bash, which comes included with Git for Windows. Our cloud structure runs on CentOS.

Netflix Conductor is a workflow orchestration engine that was constructed by Netflix to “orchestrate microservices-based course of flows” (supply: Netflix Conductor documentation). The total characteristic listing of Netflix Conductor is positioned here, however just a few key options are:

  • Workflow and process definitions are applied in JSON format. This permits for versioning and simple assist for customized instruments to create and handle workflows and duties.
  • Duties and Employees, i.e., the constructing blocks that make up the workflows and the microservices that host them, respectively, are fully language agnostic, which permits the implementation to be finished within the language finest fitted to the duties. Libraries exist for Java, Python, .NET (shameless plug: this one is developed by me), and others.
  • The database structure is pluggable, which suggests we will select what database we need to use for Conductor. Conductor comes with quite a few ready-made plugins, supporting (amongst others) Dynomite (additionally by Netflix), MySQL, and PostgreSQL.
  • Netflix Conductor comes with an (elective) UI, which permits perception and management over workflows, like (re)beginning, pausing, stopping.
  • Your entire venture is open supply and is being actively developed and supported.

In brief, Netflix Conductor means that you can automate and handle processes that encompass a number of duties (which could be processes in themselves), which will must be executed in sequence, in a dependable, constant, and scalable method.

At my present venture, we use Netflix Conductor and PostgreSQL to run scheduled ETL processes, create rapports, obtain and add information, and for monitoring and alerting. Now we have additionally investigated utilizing Conductor to automate the provisioning of consumer accounts and different infrastructure.

As talked about within the introduction, the usual implementation of Netflix Conductor is configured to make use of Elasticsearch 6. Assist for Elasticsearch 7 is built-in, however a number of adjustments must be made to allow it. There may be documentation out there on GitHub to make these adjustments, but it surely doesn’t appear fully full. I wanted the adjustments I describe under to construct and run in opposition to Elasticsearch 7.

On the time of writing, Elasticsearch 7.17.1 is the newest model of Elasticsearch 7.

Right here’s a fast abstract of the required adjustments (all file references are relative to the basis folder):

Change ext['elasticsearch.version'] = revElasticSearch6
to ext['elasticsearch.version'] = revElasticSearch7

Take away the road embody 'es6-persistence'

Change implementation venture(':conductor-es6-persistence')
to implementation venture(':conductor-es7-persistence')

  • server/src/most important/sources/utility.properties

Change conductor.elasticsearch.model=6
to conductor.elasticsearch.model=7

  • docker/docker-compose.yaml

Change picture: elasticsearch:6.8.15
to picture: docker.elastic.co/elasticsearch/elasticsearch:7.17.1

  • test-harness/construct.gradle

Change testImplementation venture(‘:conductor-es6-persistence’)
to testImplementation venture(‘:conductor-es7-persistence’)

  • /test-harness/src/take a look at/java/com/netflix/conductor/take a look at/integration/AbstractEndToEndTest.java

Change conductor.elasticsearch.model=6
to conductor.elasticsearch.model=7

Change DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch-oss").withTag("6.8.12"));
to DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch").withTag("7.17.1"));

The usual option to apply these adjustments is to create a fork of the Conductor repository and do your work from there.

Nevertheless, I discover copying over a whole repository to vary a handful of information to be a waste of sources, so I’m following a unique method.

On construct, I clone the Conductor repository, then apply a number of patch information to make the adjustments I want. Whereas these steps enhance the construct time barely, I really feel this can be a good compromise between velocity and retaining my very own repository clear and maintainable.

Right here’s how that works:

  1. Clone the Conductor repo inside your venture listing.
  2. Make the above adjustments.
  3. Within the Conductor folder, run git diff > v3.5.1-update-to-es7.patch to create a patch file. The v3.5.1 is for comfort, to point this patch solely works in opposition to that model.
  4. Copy the patch file to the basis listing, so you possibly can safely take away the Conductor repo and apply the adjustments everytime you clone the repo once more.

You additionally have to take away any dependencies.lock information earlier than constructing. This is so simple as operating rm -f **/dependencies.lock.

To automate eradicating the lock information and making use of the patches, I created the 01.get-conductor.sh script:

You may see I’m really making use of just a few extra patches. These will not be required to run in opposition to Elasticsearch 7.

At this level, you possibly can construct Conductor. I created 02.build-conductor-server.sh for this function:

Within the part, “Placing all of it collectively,” I’ll present a number of further scripts to automate the clone, patch, and construct the method additional, together with how you can use setting variables to configure Elasticsearch.

Operating Conductor in opposition to PostgreSQL is described within the README file, as properly. Operating docker-compose -f docker-compose.yaml -f docker-compose-postgres.yaml upwill begin an extra container occasion for PostgreSQL and configure Conductor to make use of it.

However in our case, we wished to run in opposition to an current set up of PostgreSQL. Moreover that, we didn’t need the Conductor database objects to be created within the public schema, as is the default.

To realize this, we have to change the config-postgres.properties file. This lives within the conductor/docker/server/config folder. However since we’re getting the Conductor repo on every construct, we can’t change it there.

So, I created a docker folder in my venture root that may include all customizations to the information within the conductor/docker folder. On construct, I merely copy the contents of this folder.

The 02.build-conductor-server.sh script now turns into:

Our native config-postgres.properties file is just a replica of the unique, with the part on PostgreSQL modified, as follows:

By including currentSchema to the datasource URL, we make certain the Conductor database objects are created within the specified schema. The applicationName property helps us with debugging database classes if wanted.

Besides that it doesn’t work. Once we run Conductor utilizing simply these adjustments, the currentSchema property is ignored and the whole lot is created within the public schema. Not cool. I filed a bug in opposition to the PR that precipitated it.

I’ll spare you the a number of hours of looking out I did to seek out the trigger and simply current the “answer.” Together with air quotes, as a result of it feels extra like a hack. Nevertheless it works.

Within the file postgres-persistence/src/most important/java/com/netflix/conductor/postgres/config/PostgresConfiguration.java, Flyway (which handles database migrations) is configured with the road .schemas("public"), which causes any try to vary the default schema to be ignored. Eradicating this line solves the difficulty.

I created v3.5.1-fix-currentSchema-override.patch for this and included it within the 01.get-conductor.sh script.

When you’re questioning concerning the ## placeholders there, these shall be changed by the values from the matching setting variables. I am going to get to that within the “Placing all of it collectively” part. Within the meantime, simply exchange these together with your values.

As a small however helpful bonus, we enabled accomplished and terminated workflows to be archived after 60 seconds (the default), by configuring the workflow standing listener.

Simply add conductor.workflow-status-listener.kind=archive to the config-postgres.properties file.

This may preserve your database from overflowing with workflows you may by no means need to take a look at once more, anyway. And even in the event you do, the info will nonetheless be saved in Elasticsearch, and accessible on demand.

To be able to make the above work regionally in addition to in our cloud setting, we need to use setting variables to configure Elasticsearch and PostgreSQL. To be able to do that, we flip to the startup.sh file within the conductor/docker/server/bin folder. We copy this file to our native docker folder so we will make some adjustments.

We add the next traces proper after the if..fi block:

Keep in mind these ## placeholders you noticed earlier? We use the instrument sed to interchange them with the values configured within the matching setting variables.

To run regionally, we add the setting variables and their values to the docker-compose.yaml and docker-compose-postgres.yaml information, which we copied to our native docker folder. For instance, see the docker-compose-postgres.yaml file:

We use the magic host.docker.inside worth to level to our native PostgreSQL occasion, since localhost will not work from inside a Docker container.

The 02.build-conductor-server.sh script copies startup.sh, as properly.

Lastly, there’s just a few extra scripts left.

03.build-conductor-ui.sh takes care of constructing the Conductor UI picture:

We are able to run Conductor utilizing 04.run-local.sh, which actually would not do greater than name docker compose up:

Or, we will run 04.run-local-postgres.sh to run in opposition to our PostgreSQL database:

There you could have it. Utilizing the above scripts, it is best to have the ability to construct and run a neighborhood occasion of Netflix Conductor in a couple of minutes. And hooking up the scripts to your favourite CI/CD instrument ought to be completely doable, as properly.

Operating Netflix Conductor with Elasticsearch 7 and PostgreSQL takes a bit of labor, however by utilizing the ability of git diff and bash scripting, we will make the method simple, repeatable, and useable in each a neighborhood and a cloud context.

All of the above supply code is on the market on GitHub.

I hope this text helped you get began with Netflix Conductor utilizing Elasticsearch 7 and PostgreSQL.

Glad coding!

More Posts