Visual Integration Patterns
I’d like to know what people think on this subject.
Right now, today – there is a certain degree of expection that Business Processes – which are typically quite “step-by-step” – have a visual tool that allows (non-)technical creators/editors/debuggers to Create/Edit/View them. This is fine for Business Processes – their “flow” is typically represented as a “step-by-step” process, with some basic conditional routing. Importantly, typically in a Business Process, all the possible routes are pre-determined. At this level, we can focus our attentions at the service-level – we dont necessarily care where (or how) each service is invoked. So the BPEL “routes” are relatively straightforward in design. Which lends itself well to a “Drag and Drop”-style GUI tool.
So I find myself recently asked a similar question about (Enterprise) Integration Patterns. Because the very nature of these messages are much lower level, they have a much greater ability to create more complex flows. They are concerned with much more “physical” attributes, compared to a Business Process. An Integration Pattern typically DOES care “which server”, “which protocol”, etc. The real problem comes with the complexity of the “flow”. When the flow is conditional, and the possible routes are NOT explicitly stored in the flow, then it becomes much harder to represent this visually. I refer to patterns like Routing Slip, or where custom Java-DSL routes are defined that internally decide how to continue to route a particular message. How can we draw a “chart” of the flow of data through an Integration Pattern, if we don’t know up-front what the flow is going to be?
I’m of course picking on the more difficult EIP’s rather than generalising, but you see my point, right? What good is a tool if it can only (visually) represent some of the patterns?
Despite this, there are already efforts to visually represent Enterprise Integration Patterns. I’ve seen examples of ESB design tools in what was Oracle SOA Suite (not sure what exists today after the BEA/Oracle technology merge). Although the scope of this was somewhat reduced – I certainly didn’t see many Integration Patterns represented in there. More recently the Fuse product team have created the Fuse Integration Designer – which aims to visualize Apache Camel routes, and include other components such as CXF and Spring-based beans. Finally, more recently, the Sun team have been busy working on tooling for Project Fuji – which also looks promising.
But my question is this – how far can this realistically go, and what will it really be used for? How important is visual tooling for Integration Patterns. I can see the benefits to the developer – the ability to quickly “mock up” 50-80% of an EIP project visually clearly saves considerable time. But once that route includes non-visual aspects that affect the route (components that cannot be visually represented) – how useful is it then? Can it still be edited? Debugged? If we are debugging, and the runtime suddenly dives into a pattern we can’t represent, and then crashes – how valuable is this?
My main concern is that it will encourage “sloppy patterns” – patterns that avoid using non-visual components, because it’s easier to stick to the components that can be created and edited in the visual editor.
Or is there really an answer that enables all components to be represented visually? The only solution I can think of is to split EIP’s into “stages” delimited by endpoints. So every single start/end route combination has a separate view. Then a design/edit/view/debugging tool can “jump” from one diagram to another, without losing the context. Clearly for internal java-based errors, we will still need to jump into the code – but it might enable better web-based “first line” debugging to take place to focus the problem onto a particular EIP “stage”, or better still, a particular component…
Thoughts?
Tagged as Apache, Enterprise Integration Patterns, Fuji, FUSE, IDE, Java
Categorized as Integration
There's always going to be limitations of what you can do in a purely visual manner; a route defined in FUSE Integration Designer is going to be part visual and part code (be it Java, XQuery, XML configuration or whatever).
That doesn't much matter in and of itself; with FUSE Integration Designer you can visualise the patterns so you get to see the big picture easily; you can then visualise/run/debug the route and step into whatever level of detail you require. You can always just debug the actual low level Java code too if you want; being open source the whole of FUSE comes complete with source code.
I've never really believed in the pure visual CASE tool type stuff of the 80's; the futures going to be hybrid of code/scripting and visual abstractions – hopefully in FUSE Integration Designer we let you get the best of both worlds; code/scripting and visual programming/visualisation/debugging.
Great – a very reassuring answer – thanks James! After giving the subject further thought since I posted, I think that, if I’m honest, my original question was being driven somewhat from a more contentious angle – which is the age-old battle of “developer vs. designer”. Once a particular technology gains a visual editor, the perception is that it (apparently) then becomes possible for “less technical” (whatever scale that is measured on) people to create components in that technology (be it application code, business processes, or as in this post – Integration Patterns). However, this only really holds true (if it does at all – I remain sceptical) if the visual tool can completely replace the existing development environment/process, and make the whole lifecycle “less hands-on”.
In the case of Integration Patterns, and this was also my reason for posting originally, I don’t think that this would ever be the case – and you also concur with this – which is very reassuring. There are many aspects of developing integration components (including patterns), which are unlikely to ever be represented visually. As you suggest, I also feel it will likely be a hybrid case where both visual and code tooling may be required to create and maintain Integration Patterns.
I take much greater comfort in knowing that a visual tool (in this case for integration) exists to improve the efficiency of existing developers, rather that trying to “simplify” the whole experience purely to support a (potentially) unsuitable audience.
Part of my main reason for exploring this topic, is that, when teams are asked to design an “integration platform”, which is to be sold to a customer (as part of a larger solution) and may be used by the customer to extend/customise their individual system, it’s important to identify who our target developers are. Who needs to be able to understand our integration patterns, and extend/create/maintain them if necessary?
In some cases, a customer may already invest explicitly in integration – they have on-site developers who currently write point-to-point or other code to allow systems to interoperate. They are likely already familiar with Java and the necessary tooling to code in this way. In other cases, the customer may be completely new to the concept of an “integration platform”. Maybe they don’t have integrated systems today, and the “integration platform” is empowering them to achieve tasks that previously they could not do. For them, it’s a “black box”. For these guys, the team needs to decide whether to (1) mandate that the customer invest in the necessary “hands on” skills (Java, integration) in order to extend EIPs in such a platform, or (2) whether the team can realistically provide them with tooling that allows this without needing to be “code-savvy”. My argument has always been for (1), and reassuringly your comments seem to agree. But the requirement for (2) lingers on, as it always does….. :)
Thanks again.
I'm lucky enough to have the job to make FUSE Integration Designer work for everyone :)
Like James, I witnessed the CASE stuff, followed by various 'workflow' technologies, followed by BPEL and now the various BPMN, XPDL, jBPM, etc. Many of these types of tech were created to lower the level of distrust that line-of-business people had for developers – who behaved differently to them, spoke in weird tongues and produced blobs of unintelligible code. This level of distrust has waned, fortunately and with the *BP* efforts you now see these tools being used as formalizations that may have no direct mapping to implementations. These formalizations make it easier for the customer/process creator to communicate their intent in a semi-algorithmic way to the development team.
Enough of the sweeping generalizations. I expect anyone using the graphical whatsits of FUSE Integration Designer to have a level of code understanding of the underlying runtime and how it works. I expect them to want to get in there and write some Java/Scala/Groovy to enhance their route. What we do in the tool is provide 'breakout' options – if you know what you are doing you can get right in there and use any URI as an endpoint, or hand-craft a piece of raw Camel XML configuration that the tool will inject into the overall route without any questions.
The motivation for the graphics are threefold –
(1) Communication. EIPs were designed as a communication mechanism – here I'm paraphrasing Gregor Hohpe – rather than a process language. If you can draw an integration solution with the EIP conventions, then it's a powerful way to communicate a useful view of the solution.
(2) Debugging. If you create an EIP solution, and messages go missing, you don't know straight off the bat what's happened. You just know the message has gone. You need to debug it somehow, but you don't want to necessarily debug Java code – you just want to see where the message vanished into the ether. If you can step through the EIP solution, using the graphical representation for setting breakpoints, you can see where the bad stuff is happening fairly quickly. Then you can drill deeper if you need to.
(3) Everyone loves pictures! Seriously – this is important. We have a great big chunk of brain at the back of our heads that is dedicated to visual processing and we focus on pictures and colourful objects. We show FUSE Integration Designer route diagrams to our field staff and they love it, we show it to customers and they love it too. It makes a big difference to the initial experience of the integrationware stuff we do, as usually demo'ing integration/middle-ware is like demoing intestines – you can't see the magic, just the output, which doesn't look that attractive.
Just a final few words on your "requirement for (2)" – it's one of my normal assertions that you can never have a totally specified business system at time X that still works without any requirement for extension at time X+Y years. There's always a need for change. So every system that you put in needs some plasticity – soft spots where someone who knows how the system works can extend it, tweak it, provide interceptors, that kind of thing. You should teach the customer how to work with these soft spots, for example giving them a tuned DSL, or a set of bindings/APIs to a scripting language.
More great comments – thanks Oisin!
Your comments raise a great architectural point, especially when concerned with using an integration platform (like FUSE) to build another integration platform (specialised for a specific domain). When two teams (the platform team and the customer-based team) are both developing at the same “layer” in the system (e.g. both writing integration code/patterns), it’s often easy to forget that we have the flexibility to offer a different development environment to the customer, to the one we use to develop in ourselves.
Who says that the platform team can’t develop their patterns in Java, while allowing the customer to extend and add patterns that are written in other languages (like Groovy, Scala, XML, etc)? I’ve long been a promoter of the “right tool for the right job”, and clearly there is great scope to apply this here. Furthermore, it doesn't necessarily mean we are then forced into using different development environments either. We have the ability to promote the same development tools (like FUSE Integration Designer, Eclipse, etc) so that the view across the two sets of components – platform and custom(er), can be viewed together.
If teams find that the requirement does persist for (2) in the post above, then it’s a question of writing platform integration patterns that can be extended (and possibly even modified) without requiring that the actual source code be directly modified. Externalised configuration, aspects, interceptors, etc all exist for this purpose, and should be leveraged as much as possible.
Brilliant – this whole "thread" is starting to look alot brighter in my mind.
Thanks again.
And finally – I hope you don’t mind – but I’ve quoted you as my new “Quote of the Moment” – the effect of reading your analogy of demo’ing integration/intestines echoed around the somewhat quiet office. It is so true :)