Wednesday, October 1, 2008

SOA, EDA and MPI.NET

I had another good question from Miguel, this time referring to my post on SOA and EDA where he asked whether distributed systems built using MPI.NET conform to the SOA or EDA style of architecture.

MPI.NET is a platform for building a single distributed application where different parts of the application run on different nodes within a cluster of machines. Often, the same program runs on different nodes, but takes on different roles on each node. The role each program instance takes on each node can be determined by a unique rank assigned by the MPI environment to each MPI process.

Messages can then be passed between different processes to coordinate their activities. Often the program detects if it is rank zero and if so takes on the "root" responsibility, handing data and control messages to the other nodes, coordinating their activities and aggregating their results.

The message exchange patterns offered by MPI.NET are point-to-point, all-to-one (gathering data), one-to-all (broadcast) and all-to-all. These message exchange patterns differ from the message exchange pattern inherent to EDA, publish-subscribe.

With publish-subscribe, systems subscribe to specific topics or topic specifications. Messages published onto a given topic are routed to all subscribers that have subscribed to a matching topic specification. Alternatively, subscriptions may be defined based on message content-based rules.

With MPI.NET, there are no subscriptions as such. Messages may be broadcasted to every node, which then decides whether the message is relevant. This is a subtle difference, and perhaps one might argue it represents a highly simplified form of EDA.

With regards to SOA, it is extremely bad practice for services to share the same business rules or data representation as it couples services too tightly. The fundamental principle of MPI.NET is that the same logic can/will run on multiple nodes. The purpose is quite different from SOA.

With MPI.NET, you are distributing execution of the same logic across multiple servers for the purpose of increasing performance. With SOA each service is responsible for its own logic, encapsulated behind its service boundary.

Furthermore, the number of nodes allocated in an MPI.NET application is dynamic by design. With SOA, each service has specific rules and responsibilities. The number of services cannot be dynamically increased or decreased.

Certainly within a service the number of nodes allocated to handle messages could be dynamically adjusted based on load. However, these nodes are not directly addressable. They are not visible to the consumers of the service.

Another difference is that MPI.NET nodes do not have the concept of endpoints. One cannot deliver messages to an MPI.NET node across a variety of different bindings via different endpoints.

MPI.NET should be seen as a way to implementing a single service internally across multiple machines for the purpose of improving performance.

I would say therefore that MPI.NET is not a platform for implementing SOA. It could be considered as supporting the EDA style of architecture, but only in a very simplistic sense.

8 comments:

miguel said...

Bill: on about the fourth reading I believe I'm starting to get it - I hope - I think. More on this later.

Nickysam: I appreciate the sense of purpose implied in your comment! (If I'm reading it right). I find that as an ex Engineering student who did it the hard way I gain more confidence with a concept when I can break it down to first principles and build it back from accumulated knowledge (usually maths...). I do, however, as an IT professional who aspires to delivering useful and robust software/systems, appreciate that focusing on the arcane or ambiguous is not the most fruitful path. That drinking the Kool-Aid and getting up to speed on "best-practice" where it is delivering real and perceived value is more useful than debating intriguing ideas. Unfortunately, in lieu of a guide/mentor I find that I must battle these concepts in my own mind and try to expand the knowledge I have by trawling the internet (and like avenues), often stumbling (in the hope of improving my ability to deliver robust useful software) :)

Bill, Ok, on about the fourth reading I think I get it (maybe).

Technology alone is not a sufficient, though it is a necessary, condition for the types of concepts you/we are dealing with here (yes/no/maybe?)!
Development/Organizational styles, such as SCRUM, Capability Maturity Model Integration, XP, et al are not necessary but highly desirable conditions, on the back of which you can start to collect metrics about organizational/develop performance.
It is the ability to decide on the right technology, that leverages an Architectural view of the world (so EDA is concerned with subscription to fine/finer grained concepts than wholesale broadcasting, and SOA is concerned with guaranteeing service orientation throughout the enterprise - not one or two web services alone) to deliver value as mandated by an organizational model that supports these activities and is capable of proscribing the correct(correcter??) pathway to achieve successful change.

Anyhow, cheers for the post and you can trust that I really did at least attempt to take it in

Bill said...

Hi Miguel,

Yes indeed technology is a means to implementing an architecture.

However SOA and EDA are not architectures, they are architectural styles.

So we may employ many and varied technologies resulting in a particular architecture, which may conform to one or more architectural styles.

A good comparison is OOP. OOP is a style of building individual software systems. Comparative styles are procedural programming and functional programming.

Comparative architectural styles to SOA are REST, EDA, client-server and mainframe. None of the above architectural styles mandate what technologies will be employed in their implementations.

Each architectural style carries its own traits. SOA specifically states that the architecture is made up of services which exchange messages via their endpoints in conformance to their service contracts.

Other than that, it is up to the architect to determine the quality of the architecture and the technologies in which it is implemented - regardless of the architectural styles to which it conforms.

Cheers,
Bill

gminorcoles said...

SOA is meaningless. Don't worry about trying to wrap your mind around it. If you have an engineering background, I am sure you will grok MPI, and grok Corba, and grok useful APIs that descend from COrba like ICE from zeroc. And you will see why, for example, WCF sucks, web services suck.

Bill said...

Hi gminorcoles,

You are unfortunately mixing your concerns here. SOA is an architectural style, independent of any technology such as Web services.

While your well articulated point that WCF and Web services in general suck is interesting, it has no bearing on SOA.

In fact, SOA can be implemented over CORBA. So since you seem to be such an avid supporter of CORBA, SOA may yet hold some value for you.

SOA is not the tool for every job. Certainly if you want to do massively parallel computations, SOA is not appropriate. You would use something like MPI.NET for that.

Cheers,
Bill

gminorcoles said...

I am sure I am guilty of some pretty severe imprecision in my abuse of terminology. But I have seen the initials SOA evolve from the "enterprise XML" craze of the early 2000's, which I lived through. You may say SOA is a design style rather than a buzzword layered on top of a bunch of WS-* standards but I think I am right. I never saw a project that tried to wrap "legacy services" in SOA succeed unless it was a POC to show that it would work in a toy app, like screen scraping a mainframe gui to a web service. The real projects, like the one where all the business users were tricked into believing that their very complex Lotus Notes ecosystem was "old" and thus somehow crusty and stale, and that web services were fresh, and thus better, resulted in a massive project that cost milions and resulted in lawsuits and ended up with an extranet app that was all XML and servlet based that never ever worked. The Notes system just keep plugging away. I am no luddite, but none of these ideas should have survived past 2001 or so.

miguel said...

Hi gminorcoles,

Thanks for your earlier clarification or earnest appraisal :)
Its not something to tell the clients I guess but between geeks I like boiling things down works/doesn't work rather than bussword driven conversations (not that we do that around here I hope).
Also thanks for putting me onto ICE I've been looking/playing with since.

I think there is something you're failing to mention though, and I completely agree that pushing management to buy into a new product just to spent the departmental pot is not productive and more often than not disruptive and detrimental (as in the case you proposed). However there are uses for WS-*, or CORBA or XMLRPC(i guess) or anything that ACTUALLY GETS THE APPLICATION WORKING, TO SPEC AND FIT-FOR-PURPOSE ( i don't think you disagree btw) and there are very large problems that meet a collective set of parameters/constraints and which need to develop reproducible and testable software solutions for, so my question to you is if not SOA what?

Thanks,

Miguel

Bill said...

Hi gminorcoles,

Unfortunately there are many who have experienced SOA project failures. In fact, most SOA projects that have occurred to date have not met expectations.

This is not however a result of SOA. It is a result of poor architecture.

Just because an architecture is service oriented, doesn't mean it is of high quality.

A good parallel is with OOP. I've seen my fair share of extremely poorly designed systems written in an OO language.

In the OO world we now have well recognised design patterns and principles which we can reuse in different contexts as guidance for good designs.

Very little such guidance exists in the SOA world. In fact an absence of guidance would be better than what we have today. At the moment there is a lot of bad guidance that seems to get perpetuated in the absence of anything better.

The hype around SOA hasn't helped at all. The hype was primarily triggered by the eventuality of Web services, which at first were very RPC rather than message centric.

The original design principles evolving around SOA were consumed by the XML/RPC style thinking embedded in the original Web service standards and surrounding guidance.

The hype around SOA has also inspired many vendors to push their own agendas, redefining SOA to suit their specific middleware stacks.

Recognising the lack of available sound guidance around SOA is the reason why I started this blog.

In time, quality guidance will become more readily available and we will begin to see successful SOA projects become mainstream.

Personally, I have had a great amount of success with SOA in my commercial projects based on the principles espoused in this blog. These projects were not proof-of-concept projects. They were at an enterprise level affecting business critical production systems.

Cheers,
Bill

Firoz Khan said...

Restaurant Wifi-Get Free Social Wifi, Facebook Marketing Strategy, Social media tips & tools for better business management & strategy with your WIFI by from VizzWifi.co.uk.