Saturday, April 19, 2008

Web Services Integration

As I've recently discussed, SOA and EAI are two fundamentally different concepts guided by different principals and objectives. However, EAI practitioners often will leverage SOA (as a style of architecture) as a means to achieving their EAI objectives.

In this case, EAI practitioners will not interface directly with applications they are attempting to integrate. That is, they will not leverage direct database access, file transfers or RPC APIs in integrating applications. Rather they will leverage only Web service interfaces as the means of integration with all applications.

Where the desired functionality is not exposed by existing Web service interfaces, a service wrapper is applied around the application, which acts as a layer of abstraction between the application implementation, and the desired functionality to be exposed.

This practice is known as Web services integration, or otherwise service oriented integration. This approach offers some improvements over traditional EAI in that all applications speak the same language (SOAP), and that we have looser coupling as a result of encapsulating application internals behind service contracts.

Despite these improvements, Web services integration still suffers from the fact that it is a bottom up approach. That is, we are focussed only on integrating existing applications without any thought going into the design of the applications themselves or how to align our services with the business they support.

As such, just with EAI, we end up with sub-optimal distribution of data and functionality across the applications being integrated. This then results in a lot of chatty synchronous CRUDdy RPC-style interactions between applications, and a lot of heavy lifting to be done by middleware components.

This in the end means we have tighter coupling between services, which impacts our ability to safely and easily make changes and also negatively impacts our overall performance and reliability.

Unfortunately, Web services integration is the vision being pushed by most vendors today, and as such forms a large part of the documented "guidance" on SOA available. Consequently, Web services integration is the most commonly practiced approach to SOA, and is the image most people get in their minds when they think about SOA.

In order to get a better quality SOA, we must align our services with the business, expose process centric interfaces, rely mainly on publish-subscribe messaging and decentralise our data.

7 comments:

Graeme Foster said...

Hi Bill,

How would you go about exposing service publications from existing applications which just expose custom APIs.

I can imagine a service (biztalk / custom app / whatever) which would observe the application (using its API) and then raise events when necessary.

Is this how you would see legacy apps participating in a Pubish / Subscribe manner as opposed to just exposing Command style web services?

Thanks,

Graeme

Bill said...

Hi Graeme,

Some applications do provide a mechanism for getting at events within the application.

For example, Microsoft CRM provides a "callout" facility whereby you can provide a .NET assembly that conforms to a specific interface that MS CRM will invoke when certain events occur.

This kind of facility is not always available however and in these cases we need to sometimes get a bit creative in looking for other ways for observing events.

In essence, if we can't observe events via the provided APIs, then we really need to go directly to the application database.

This is achieved by applying triggers within the application database which then write to an event log. We then have a separate process that polls the event log on regular intervals and publishes any recorded events.

This approach is of course not ideal because when the application is upgraded we need to potentially revisit the integration logic as the database schema will likely have changed.

This is not a major issue however as it only affects logic behind the service boundary and as such, other services in your enterprise are shielded from the change.

There are some legacy applications out there I am sure where you simply just can't detect or observe certain events in any way whatsoever - although I am yet to come across one.

In that situation, you simply would need to make do without including those events in your event catalogue, which certainly would impact your overall architecture.

If you think creatively however, there are usually ways and means of detecting these events.

Graeme Foster said...

Sure if the application does post events then you would hook into them. But many legacy LOB applications wouldn't have had the foresight to do that.

Going directly to the database sounds risky. Maybe it's just a knee-jerk reaction but I've seen too many crazy database schemas which leave you in a world of voodoo and black magic trying to understand them!

And from a customers point-of-view wouldn't adding random triggers to a third party applications database have implications around support contracts, etc.

Bill said...

I've not found it to be an issue in the past myself.

You're right - it isn't pretty. But then legacy applications rarely are.

If there are existing support contracts in place, that in fact works for you because you should be able to rely on those to get the modifications done for you.

Also remember, the changes you make to the legacy DB only become problematic when the legacy application is replaced or upgraded.

If you are replacing the application, then you can choose one that exposes events natively. If you are upgrading it, then it is possible that the new version of the software will give you a mechanism for getting at events.

Needing access to application events is not an uncommon requirement for integration and as such a number of modern applications will give you a way to achieve this.

Dave said...

Hey Bill. Great posts!!! Good timing considering we are involved in some SOA vs EAI comparisons and it isn't easy to compare.

One qustion - Vendors such as Software AG/Web Methods preech that SOA is also about legacy modernisation. What you have posted here regarding Web Service Integration apparently also could mean legacy modernisation. What are your thoughts on SOA provides legacy modernisation? Some people think that this is a key tenets of SOA.

Bill said...

Hi Dave,

Thanks! It's great to hear that the posts are relevant and helping people out. :-)

There are many benefits that vendors claim will come from SOA. These include increased reuse, greater visibility, business empowerment, increase in business agility, as well as legacy rejuvenation/modernisation.

In many respects, a number of these benefits overlap and all to some extent contribute to business agility. This is why it is said the business agility is the core business driver for SOA.

Legacy modernisation is about taking existing applications and having them participate in an architecture which is beyond their original design and adds more value than was originally intended.

SOA certainly can achieve this. And if done properly, we can have an architecture that is very loosely coupled and drives significant business agility.

As part of this endeavour, we modernise our legacy applications, which is a very nice side benefit.

Legacy modernisation as an objective on its own however in my really smells like EAI. With SOA, integration is a by-product - not the primary objective (which should be business agility).

SOA in the short term is more expensive than EAI, but in the long term has considerable cost savings due to gains in business agility.

Sometimes it can be difficult to justify the early cost of SOA if application modernisation is the only strategic objective management is pursuing.

aparna john said...

Hi,Every web designer knows a client should be asking, often go unasked and unanswered unless your Web Design Cochin firm brings these matters up, but sadly, many do not.Thanks..........