Pragmatic Legacy Application Rejuvenation

In my last couple of posts, I've focussed on how we go about leveraging existing legacy systems as part of transitioning an organisation to SOA. Just to reiterate, we first identify and define our services based on the business context, and then determine which existing applications fit into each service. We then apply a layer of abstraction between the service boundary and boundaries of the applications held within.

However unfortunately this is not always possible. Legacy applications unfortunately are existing IT assets that act as constraints for our SOA. Despite there being many ways in which we can attempt to expose events from legacy applications, sometimes this is simply not possible due to their design.

In these situations, we must unfortunately compromise on our service definitions. We should however consider the ramifications of this. Compromising on a single service contract can impact the design of many other services. This long term cost should be weighed up against the short term cost of replacing the troublesome legacy application sooner rather than later.

Publish-Subscribe with Legacy Applications

Graeme posted a couple of questions regarding my recent post on Web services integration. Specifically he was asking how best to get access to events in third party applications that do not support any native mechanism for hooking events that occur within the application.

This is a good question as it is a common problem to overcome. The fact of the matter is that when we are trying to leverage third party applications as part of our SOA, we are constrained by the limitations of those applications. If an application does not provide a native way of getting at events, we have no choice but to extract the events from the application database.

As Graeme points out, this is less than desirable as the database schema of an application is subject to change. The application vendor will likely give you no guarantees about the stability of the database schema, nor warn you when it is about to be updated.

Although this is indeed true, I don't think it is sufficient a problem for us to avoid publish-subscribe. Anytime we upgrade or replace an application with which we integrate as part of a service, we may have to do work to cater for changes in how we integrate with that application. This problem is not limited only to hooking events.

So what techniques can we use to get at events from the database? Well whatever works really. One way we can go about it is to apply some database triggers to write into a log table whenever something changes in the database. We can then poll that log table and publish events based on entries found there.

Another approach is to add a nullable timestamp column which is updated by a trigger every time a row is inserted or updated. We then poll the table for records with a timestamp after the last time we polled.

Some applications make getting at events from the database quite easy as they keep some kind of historical data. So for instance, a CRM application may keep a record of changes to the customer for audit purposes. In this case, we would just poll the audit table for any new records.

The effort required to hook these events in my opinion is certainly worth it. Events in SOA are highly reusable. So the fruits of your labour can be reused across a number of services. It also enables a decentralised data architecture, which considerably reduces coupling between services.

Legacy Application Rejuvenation Example

Badis recently asked a good question regarding ways of exposing functions which are implemented by existing applications as part of an SOA:

...I am new to SOA and I work for a group who owns 5 different large web applications. These applications, although, all Java-based, use different technologies. Some are implemented with EJB, others with Spring or home grown frameworks. What is the least-invasive approach to expose these functions as services in your experience.

Before answering this question, there are a number of things that need to be explored. Firstly, I want to just raise a flag regarding terminology here. We do not expose functions within an application as services. A service in the context of SOA is considerably more coarsely grained than a single function within an application. Usually we will find entire applications sitting behind service boundaries.

Before considering how to expose functions in an existing application as part of an SOA, we first need to define our services. Services should be designed from the outside in, not the inside out. That is, we need to understand the service's place in the world before figuring out what goes inside. We want to try and stay clear of a bottom-up Web services integration approach, where services are defined based solely around our existing applications.

Each service should support one or more cohesive business processes. Usually, existing applications do no such thing. They usually embody a number of loosely related business processes that were all brought together to serve a particular purpose at the time the application was built.

Our services need to be identified and defined such that they are aligned with the business. This is the only way we are going to get high cohesion within services and loose coupling between them. Once we have defined our services, then we can then determine to which of these service(s) each of our Web applications belong.

We may find that all five Web applications actually are concerned with the same single service (although this would be unlikely). For example, we may have five different CRM applications for dealing with different types of customers. In this case, all five applications would belong to a single Customer Management service. Although we wouldn't really have an SOA until we added more services. A single tree doesn't make a forest.

When defining our services, we'll very likely be making heavy use of publish-subscribe messaging. In this case, we will update and/or integrate with the existing Web applications not to expose functions as such, but rather to publish events and process event messages when they are received.

We need to evaluate the semantic/functionality gap between our service contracts and the Web application(s) inside each service boundary. Where the gap is very wide and/or we don't want to or can't directly modify the source code of the Web application(s), then we would consider using some kind of EAI tool. Where the gap is very narrow and we are happy to modify the application source code, we can update the application(s) to sit directly on the service bus.

When updating an application directly, which is the situation Badis asked about, we implement a "service interface layer" as part of the application. This layer sits on top of the "application service layer". The application service layer is shared by both the service interface layer and your UI logic (i.e. your controllers in the case of MVC). This allows us to expose the same logic via the UI as well as to other services.

Note that an application service is very different from a service in the context of SOA. I've spoken in the past about how horribly overloaded the term "service" is in IT architecture - this is yet another example.

The service interface layer consists of a number of message handler classes (one for each type of message the application must process) as well as event classes (one for each type of event message the application publishes). This layer would be implemented using the same technology as the rest of the application (EJB/spring/custom).

The application needs some means of receiving messages off the service bus. In essence, the message handlers need to be hosted somewhere. They could be hosted in a Web server (if we are using the HTTP transport), or alternatively you could host your message handlers directly inside an ESB.

In essence, each message handler will be invoked by the infrastructure once the appropriate message is received. The message handler takes over from there.

Sharing Applications Between Services

A reader posted a good question regarding my Service Boundary Example post. He asked:

Let's say as a result of our enterprise architecture planning we've decided that we're going to have a Sales Service (Leads, Opportunities, etc) and a Marketing Service (Campaigns, Promotions, etc).

However let's say we currently have one legacy CRM system that supports the marketing department and the sales department.

Which service boundary will contain that system?

The answer to that is that different parts of the legacy CRM would sit behind each service boundary. There is no issue in sharing infrastructure between services. You just need to be aware of the dependencies on that infrastructure from a risk management perspective.

Service boundaries are drawn up around data. There must be no data shared between services except through explicit message exchanges.

At the very least I would suggest that we have process isolation between services, so that data isolation is enforced at the OS level. To be fair however, you don't really require process isolation between two services in order for them to function as services. It is just obviously a good design choice.

So in this case, as long as the lead and opportunity data are never used by the Marketing service, and the campaigns and promotions data are never used by the Sales service, then we have data isolation and we haven't offended the SOA gods too deeply.

When transitioning legacy systems into an SOA, we sometimes have to compromise further on this because we have a single legacy system used by two services that doesn't enforce this data isolation. We can't control the design of this system. And to be fair, it wasn't designed to be used in this way.

The issue is we now have a message sent to one service, affecting the state of the second. This is obviously very bad and not ideal, and I wouldn't consider these services as true isolated services in the context of SOA. However it can be a useful half way point on the way to true SOA.

Service Boundary Example

After my last post, I thought I'd give an example to clarify the point I was making about service boundary definition. Let's say we've got a couple of legacy custom built CRM applications that we want to leverage in support of a new CRM service. Let's say one application is used for VIP customers, and the other for regular customers.

We have identified that we want a single CRM service to support the Sales department. The development team decides that they will expose Web service endpoints on each of the legacy CRM applications in order to provide integration points for our new CRM service.

We define our CRM service contract based on the business function we want our service to perform. Note that we now have 3 service contracts, but only one service. We have the public CRM service contract, which is what defines our service boundary; and then we have a service contract for each of the legacy CRM applications, which sit behind the service boundary.

These legacy CRM applications are not services in the context of SOA, despite accepting messages over a WS-* stack defined by a WSDL document. They are an implementation detail of our single CRM service, and should not be directly accessed by anything outside the CRM service's boundary.

We now need to implement the the public CRM service operations, leveraging the private operations exposed on the legacy CRM applications. We might employ a tool such as BizTalk for this job.

At a future point in time, if our enterprise decided to replace the legacy CRM applications, we could do so without updating the CRM service contract - assuming of course that the required functionality offered by the CRM service to the enterprise did not change.