The Remote Engine
I’ve spent a lot of time consulting and implementing BPM solutions for customers. In those projects, an embedded process engine was almost the default go-to choice for integrating the process orchestration capability. I even made a post extolling the virtues of this option. Why? Because it was one of the best options we had at the time we made those choices. So what changed? Not much. It still is.
Camunda’s announcement that their cloud-native process engine Zeebe, combined with re-imagined tools to make for a complete process orchestration solution has brought something completely new to the table. It’s not as if this came falling from the skies, mind you; the project started out over ten years ago. You could say that the kind of tools in the general ecosystem for cloud native application development have also made a lot of the parts available that are required to make Camunda Platform 8 a reality. Process orchestration is no longer limited to squeezing out every last drop of hardware and software performance to keep up with its environment that has surpassed its scale. It can keep up by using the same principles of horizontal scalability and TODO.
Camunda’s product positioning aside, this did get me thinking about the primary alternative that this new offering puts front and center: a remote engine. This post investigates the merits.
Before we already had and used this option. Technically it would could always be considered an embedded engine because it was just a library added to a Java application and ran inside of that application. Even if that application actually did almost nothing else. What makes it remote then? I guess you could say as soon as you separate the runtime of the process engine from that of the core business logic, which the engine is supposed to be orchestrating, you have a remote engine scenario from the perspective of that application.
Initial Thoughts
I have to admit I needed to overcome so initial cognitive bias with the term remote engine. It brought back memories of centralised process orchestration tools that were slow, heavy on performance requirements, monolithic, obtrusive and inflexible. Needless to say, those associations weren’t positive. The embedded engine pitted against many of those characteristics by being flexible, nimble, and focussed to the point where you could add process orchestration to an application and get going and growing. There were limits, though.
Thread pools Polyglot Database
The EE was cloud ready, but not cloud native.
Trashing on Embedded Engines
I don’t even agree with all the points Bernd mentions in his post link as disadvantages of the embedded option.
Not the form of embedded that really counts, it’s the capabilities that you add to your application. Let’s revisit the main advantages I listed.
You can definitely optimise with remote. even within your service bounded context! With embedded you mostly had one engine config.
A lot of config had to do with the engine in your app. Bernd doesn’t even mention the database upgrades.
Still work to do: low volume application s leveraging cloud scalability but mainly cloud elasticity!
Take the idea of remote application to the extreme: let’s make it infrastructure. Process orchestration galore: just use it.
Notes from Bernd’s Posts
-
Conflicting thread pools with the embedded engine
-
Why bother fiddling around with the thread pool just to get the engine to do its job?
-
-
Optimize per service or application or even process is still possible - you don’t HAVE to have one remote engine!
-
Also, the application can now determine itself how to scale and make each part performant, since external task patterns are the default.
-
-
Class not found: delegate
-
The embedded engine may have been cloud ready, but not cloud native
-
The engine in your application is not what you need, the state machine is, tracking and instructing on the way. The interesting part is still what happens in each step, which is in your app.
-
Is it functionality to have an engine or is it simply infra?
-
BR says that debugging is harder when embedded: really? How about troubleshooting distributed systems?
-
@ BR: data model updates is also a disadvantage of embedded engines!
Glue code article: * 1 worker app, also deploys the models.
Own notes
Embedded: libs and deps are a hassle - even though there are not many from the engine, but every framework brings a few… clash! Embedded: db upgrade is now tied to application
Downsides: * Complex, distributed system. It’s this nature that not only brings many of the scalability benefits, but it also creates problems.
camunda/20220323/DM/6: platform like deployments due to separation of engine and application. Even pure remote C7 is an application, because of wrapped up in spring boot. camunda/20220323/BR: embedded engine calling the glue code or the embedded client calling your glue code.
Remote: centralised resource, so easier to roll out in new areas without all the hassle of doing it all over again