Taking Questions Offline

I've recently received a couple of questions by way of people posting comments on recent blog posts as I haven't provided a way for people to get in touch with me directly.

That has now been remedied. If anyone would like to get in touch with me for whatever reason, you can do so by emailing me at creative.abrasion@gmail.com.

Of course as usual, any and all questions are welcome.

Enterprise Canonical Schemas (continued...)

In my last post we discussed the concept of the enterprise canonical schema and its suitability when applied in the context of the various flavours of SOA we've discussed to date. As part of that discussion I stipulated that an enterprise canonical schema should be avoided when engaging in self-contained process-centric service models.

However, canonical schemas can still be very useful when we are engaging in EAI behind the service boundary. That is, when we have one or more services containing a number of legacy systems whose activities need to be orchestrated in support of those services.

Self contained services control and hold their own data locally. Data is represented within each service in such a way to best support the cohesive business processes that execute within that service. If we were to have two different CRMs within an enterprise, both would likely sit in the same Customer Management Service.

The Customer Management Service would have its own service contract expressed in terms of the customer management business domain for the enterprise (rather than in the terms of the CRM applications within the service). We have a layer of abstraction (an anti-corruption layer as termed by Eric Evans) between the CRM applications and the service boundary that coordinates the activities of the CRM applications within the service boundary in support of the service's business processes.

So that we don't have to deal with two entirely different data representations within the orchestrations coordinating the activities of the CRM applications, we develop a service canonical schema which is aligned with the service domain model. If we have multiple services enagaging in EAI internally, then we will have a different canonical schema for each service.

Any messages outbound from one of the CRM applications are first translated to the canonical schema. Likewise, any messages inbound for a CRM application are first translated to that CRM application's native syntax. The orchestrations coordinating the activities of the CRM applications are then expressed in terms of the canonical schema.

Of course, if a service does not include a number of applications that must be integrated in support of that service then there is no need for a canonical schema at all.

Enterprise Canonical Schemas

There are many references in the available guidance on SOA that recommends the use of an enterprise canonical schema. The idea is that you standardise all data transferred between services via messages such that they conform to a single canonical schema.

The process of producing such a schema is known as schema rationalisation. It involves forming an agreement between all relevant stakeholders in the enterprise on a universal representation of all entities (such as customers, orders, etc) relevant to the business.

Any outbound messages from a service are first translated to be consistent with the canonical schema before being sent. Likewise, any messages inbound for a service are first translated in that service's native message format before being delivered.

The purpose of this approach is to decouple interacting services from the native message formats of each service and to minimise the volume of message translation logic necessary to support the architecture.

If we have n services in our enterprise, then we have up to n(n - 1) interactions that require inbound and outbound message translations between native service message formats. Through use of a canonical schema, we only have 2n message transformations that need to occur.

So this all sounds really good right? Well it depends on the flavour of SOA in your enterprise. This approach isn't really going to add much value in a JBOWS architecture. JBOWS architectures don't add much value at all, regardless of the presence of a canonical schema.

But what about Service Oriented Integration (SOI)? Well as it turns out, a canonical schema is very useful here. With this flavour of SOA, applications are exposed directly as services in your SOA. As a result, you don't really have a lot of control over the syntax of messages understood by your services.

Message schemas tend to be quite volatile with SOI as they are based on the applications in your enterprise, rather than the business processes they support. A canonical schema decouples services by shielding them from message syntax variations in other services.

Furthermore, with SOI we may have multiple similar applications participating directly as services in any given architecture. For example, an enterprise may make use of two different CRM packages, which would by extension mean we have two CRM services. Because with SOI business processes are generally executed in middleware sitting between services, when a service sends a message it may not in fact know which CRM instance the message is bound for.

With a canonical schema, this decision is not relevant in constructing the outbound message. The middleware translates the message to the native format of the appropriate recipient as the message is delivered.

A canonical schema is also very useful (in fact one may say essential) when using a layered service model. Entity services abstract native data repositories into a form which is then used by task and process services. By ensuring data exposed by entity services conform to a canonical schema, we simplify our task and process services.

So what about self-contained process-centric service models? Well here a canonical schema actually works against you. With this flavour of SOA, you've already done the hard yards of building service contracts based on cohesive business areas. Your service contracts are aligned with the business (as opposed to technology artefacts) and as such are relatively stable.

Furthermore, with this flavour of SOA each service contract has been tailored to express information optimised for a specific business area. By placing a canonical schema between services, you diminish the semantic fidelity of communication between those services, mitigating the value of all that hard work.

A canonical schema not only standardises the syntax of communications between services. It must also by extension standardise its meaning, or semantics. An enterprise canonical schema tries to build a common language to suit the entire enterprise. All discussions between services are first translated into this language.

This is very likely to be a lossy transformation. Even if you don't lose data, you'll definitely lose information through blurring of semantics.

So as long as you go down the road of self-contained process-centric services, you should stay clear of enterprise canonical schemas. However, a canonical schema can certainly add value if you find yourself in an existing SOI or layered service model environment.

Sharing Concerns between Services

Graeme recently posted a good question regarding sharing of common concerns between services:

"Let's say that a service creates a document but the enterprise as a whole has an investment in a DMS. Would you see the DMS as a centralised data store that should be de-centralised?

Most applications I've seen would just create the document and then call a WS to store it inside the DMS but that goes against the autonomous nature of the service.

Would be interested in your thoughts on this."

There are two alternatives to approaching this problem, and as usual it depends on your specific business requirements.

The first alternative is to consider the DMS application as a distributed component shared by multiple services. In this case, the DMS is not a service in its own right directly accessible by the enterprise-wide service bus. It is a private component of the services leveraging it, sitting inside their respective service boundaries. As such, synchronous request-reply communication with the DMS would be quite acceptable.

There isn't really any problem in sharing a distributed component between services, so long as data is not mixed between services inside that component. Services should share data only by way of explicit message exchange.

You just need to be aware that if the shared component falls over, all dependent services will fail.

A good acid test to determine whether this solution is appropriate is to consider whether the documents managed by each service should be isolated from each other. So if the documents produced by multiple services should be managed together (for example stored together in the same logical folders), then this indicates that this approach is most likely inappropriate.

The alternative is to have a separate autonomous Document Management Service that interacts with your other services by exchanging messages in accordance with their public service contracts. In this case, the DMS application is a private component only of this service. In my experience, this is the better approach.

This particular approach can be tackled in a couple of different ways. Assuming each of the various services in your enterprise generate their own documents, they could either asynchronously send the Document Management Service a command message (such as a StoreDocumentRequest message) containing the document, or publish a business relevant event message (such as a PolicyCancelledNotification message) with the document attached.

With the command message approach, the message would contain any relevant metadata to store against the document. With the event message approach, the Document Management Service would extract the relevant metadata from the event message body.

Although I usually avoid command messages, they are useful and appropriate when the service operation is highly reusable.

Filing a document in a document repository would be a suitable candidate for a command message. That is, its behaviour would be unlikely to differ based on the context of the service sending the command message.

If your users interface with your services via a composite application, then you can have a document management module in that application that allows your users to interface with the Document Management Service in such a way that provides a seamless integrated user experience.

Something else to consider in this area is the generation of documents. If you have software that generates your business documents based on end user managed templates, then this would be a good candidate for a component to be placed in your Document Management Service.

The document management module in your composite application would provide the ability for your users to manually generate documents by sending messages to the document generation component. This component would place the generated document in an output folder and return the URL to the composite application in the reply message. The application would then retrieve and display the document at that URL.

Yes, this interaction between the composite application and document generation component is synchronous request-reply. However this is perfectly acceptable because both the document management module and back end component are behind the boundary of the same service.

If after generating the document the user decides to store the document in the repository, then this would be a separate message from the composite application to the document repository component, which would contain the document URL and associated metadata. The document repository component would pick up the document at the given URL and store it in the repository with the given metadata.

So we have two ways in which documents can be generated and stored. One is where the document is being automatically generated by a service, where the service can send a command message containing the document and associated metadata to the Document Management Service (or publish an event message containing the document onto the service bus to be picked up by the Document Management Service).

The other way is where a user manually generates the document via the document management module within your composite application. The composite application is then responsible for sending the message to the document repository component at the user's request.

SOA Quality Spectrum

We've covered a lot of ground over the last month since I first proposed my definition of SOA, and I wanted to take the opportunity to do a bit of a recap and tie it all together by comparing and contrasting the various flavours of SOA discussed to date.

As I've previously mentioned, there are good SOAs and bad SOAs - with some considerably worse than others. So what are the typical metrics we might use in assessing the quality of an SOA? One might say it depends on the needs of the business the SOA supports. That is, an SOA is of high quality if it clearly aligns with and supports the strategic goals of the business.

Obviously every business is different. Even businesses that compete in the same market and service the same customers have unique characteristics. They have different organisational structures and business processes. Their distinctiveness is what gives them their competitive advantage. It is what differentiates them from their competitors.

Business strategy is heavily influenced by the external forces that operate on a business. Porter's five forces analysis is a good way of identifying these external forces and deriving strategies for gaining competitive advantage.

Regardless of the business and the market in which it operates, an organisation must evolve and adapt as the competitive landscape changes. The ability of an organisation to invoke change quickly, inexpensively and at low risk is a key factor influencing its success. It facilitates innovation and affects how responsive an organisation is to market forces.

Although there are many factors that influence business agility unrelated to IT, a machine is only as fast as its slowest moving part - and in many large organisations, that part is IT. For these organisations, IT becomes the bottleneck, resulting in the perception that IT is a necessary evil, always at odds with the rest of the business.

So what aspect of an IT architecture determines agility? Simply put - coupling. Coupling determines the extent to which change within one software component affects another. With a high quality SOA, a single change to how a business chooses to implement a specific business capability is isolated to a single service. As only the implementation has been affected, the service contract remains unchanged and consequently no other service is affected or even need know about that change.

So although there are other quality metrics that vary from business to business and industry to industry, agility is a key universal quality metric for an IT architecture. Based on this metric alone, I propose the following SOA quality spectrum.


I've included JBOWS, Service Oriented Integration, Layered Service Models and Self-Contained Process-Centric Service Models on the spectrum. I'd be interested to hear from anyone who has seen any other flavours of SOA out there that I've missed. I'm also interested in getting feedback from people who would propose alternative or additional quality metrics for SOA.

Layered Service Models are Bad (continued...)

In my previous post I introduced a couple of example typical business processes for an insurance company and illustrated how they translated to a service model based on process, task and entity services. After listing a number of shortcomings of this model, I promised that I would suggest an alternative service model that didn't suffer from the outlined issues.

As I previously mentioned, services should be centred around cohesive business areas/capabilities. We also wish to decentralise our data so that each service has the data it needs locally to service each request without retrieving data from other services. Furthermore, we want to rely mainly on publish-subscribe messaging in order to avoid the coupling introduced by command messages.

So our first order of business is to identify our cohesive business capabilities upon which we will base our services. Through analysis of the originally described business processes, there appears to be the following five cohesive business areas at play - Premium Quotation, Policy Administration, Commission Management, Billing and Customer Management.

Creating a service around each of these business areas gives us the following service model.


Firstly note the simplicity of this model over the model outlined in my previous post. There are only five services, and the dependencies between the services are limited to communication over three topics.

There is no need for any entity services as each service depicted in the above model houses its own data locally. Consequently we have no need for cross-service transactions.

The business logic previously contained in task and process services relevant to a given business area is now fully contained in the corresponding service. As a result, we have far improved encapsulation. Our data is protected from direct manipulation by other services.

The Policy Creation and Policy Reinstatement processes are now mapped to the above service model as illustrated below.

Policy Creation

Policy Reinstatement

Note also with this service model that we have no synchronous request-reply communication between services whatsoever. This results in a considerably more robust, performant and loosely coupled architecture. The high cohesion found within each service further contributes to the loose coupling.

We have coarser grained, business relevant messages exchanged between services and there are considerably fewer message types to deal with. And to top it all off, this service model will take less effort to implement and considerably less effort to adapt and maintain.

Layered Service Models are Bad

Open any number of books out there on SOA and you'll find guidance on defining the service model for your enterprise in terms of process, task and entity services. This in essence sets up three layers of services, with process services as the top layer, task services underneath the process services, and entity services at the bottom.

The purpose of entity services is to provide an abstracted view of business document repositories. The actual structure of the underlying data and the platform holding the data are abstracted into the form of entity service contracts.

The operations exposed by entity services are your basic CRUD centric operations such as GetCustomer, GetOrdersForCustomer, UpdateCustomer and DeleteCustomer.

Task services each represent a single atomic business action. Task services leverage entity services to manage state.

Process services are then created in support of business processes. Each process service wires up any number of task services in support of a specific business process. Process services can also "invoke" other process services when there are sub-processes to be reused by those processes.

Consider the following two vastly oversimplified business processes:

Policy Creation
This process involves the creation of a new policy. An underwriter uses a composite application to first go and get a premium quotation, and then if the customer accepts the premium create a new policy.


Policy Reinstatement
This process involves reinstating a previously cancelled policy.


Note that the Allocate Commissions and Raise Invoice tasks are "reused" between the Policy Creation and Policy Reinstatement processes.

Translated into a service model based on process, entity and task services, we get the following:


For those who follow this blog, you would probably guess that I'm dead against this kind of service model. There are so many problems with this model it's hard to even know where to start:

Transactionality: The task services must perform updates across numerous entity services. The task services are atomic in nature, which means that either the entire task must succeed or fail. This means we need to have cross-service transactions between entity services.

As I've explained in previous posts, cross-service transactions are bad for performance reasons and because they hurt the autonomy of services. One entity service can have records locked while waiting for another entity service to either commit or rollback its local transaction. One entity service can heavily influence the execution of another.

Performance: As mentioned above, cross-service transactions hurt performance. However, the other thing to notice about this architecture is that it is full of synchronous request-reply communications. This approach simply does not scale.

Robustness: As I've mentioned previously, synchronous request-reply interactions between services seriously hurts the robustness of your architecture. If one of those entity services goes down, you can pretty much say goodbye to a large number (perhaps even the majority) of systems in your enterprise.

Cohesion: With this service model, multiple services deal with very similar business concepts. You have the same entities being managed by multiple services. You have multiple task services dealing with the same entities. This means you are likely to get considerable duplication of logic and information representation between services.

This means your developers need to write considerably more code. Furthermore, the communication between services is likely to be very chatty, involving a large number of finely grained messages.

This in turn means greater coupling between services. This results in even more code to be written in order to handle the vastly increased number of message types. In essence, the proposed services in this model are too finely grained.

Coupling: As mentioned above, the low cohesion between services means greater coupling. You also get temporal coupling as a result of all the synchronous request-reply communications.

Complexity: There are simply too many services defined in this service model. With this approach, you will likely end up with hundreds or even thousands of services supporting an entire enterprise - each with dependencies to be managed and understood.

Encapsulation: By exposing data directly via entity services, any service can update any data in whatever way they see fit. This results in very poor encapsulation. We want our data to be surrounded by business logic that enforces our business rules.

Service Contract Stability: The implementation details of cohesive business processes are exposed in the service contracts of the task and entity services. As the needs of the process services they support change, the service contracts will need to be updated. I'll cover service contract stability in another post.

People are usually allured into using this kind of service model by the promise of reusability. That is in fact the fundamental premise of the architecture - that process, task and entity services can by simply reused by dropping boxes that invoke those services onto a process orchestration design surface.

However as I've previously discussed, the promise of this kind of reuse is false. It is all very good in theory, but as usual the devil is in the detail and you simply don't get this kind of reuse in practice.

In conclusion, services should be self contained; aligning with cohesive business areas/capabilities. The concerns of a single cohesive business capability should be supported by a single service. Not by many process, task and entity services.

In my next post, I'll describe the service model that I would suggest to support these business processes, so stay tuned!