The current enterprise architecture which is dominated by stand-alone vertical applications is struggling with the increased demand for integration. Each new integration is acting as a reinforcing bar that makes the IT solutions increasingly rigid, causing changes to be expensive and time-consuming.
This article gives a historical background to how the current architecture was developed. If we understand how the integration efforts have evolved over time we will also appreciate why organizations need a new architecture to meet the challenges of maturing digitally.
"Standalone" applications
The IT landscape today is dominated by stand-alone vertical applications; i.e. applications that are specialized on a specific functionality with no or little interference with other applications. Most current standard systems are of this kind; CRM, economy, PIMS and CMS systems are some examples. Human interaction with these vertical applications are done through their proprietary user interface. The majority of these applications have little or no support for guiding the user through the different tasks that need to be carried out. If a business process requires that tasks are done in several applications, the user must use different proprietary user interfaces to complete the tasks and know the ins and outs of that business process.
If the systems are actual silos, the organization is at the Nascent maturity level:
In practice the systems are not stand-alone, they need to communicate with each other. As long as the integrations are few the situation is under control. This scenario is typical for the early stages of the Emerging maturity level:
The rise of "spaghetti" integrations
The problem is that there are so many reasons to add more integrations. Some examples:
- While doing things in one system, the end-user requires information that is located in another system.
- The result from one operation in one system needs to be transferred to the next system to be used in the next step in the business process.
- Web applications and mobile applications need to read data and carry out operations in the systems.
- Things that happen in one system should trigger that something happens in another system
This has resulted in all sorts of technical integration solutions:
- Sending over files with data every night between systems.
- Making "views" into other systems from the user interface of one system.
- Mirroring data from other systems into a system that was not designed to have that data.
- Opening up the database to allow read and write operations from the outside.
These integrations are typically done ad-hoc and they are implemented based on the skills and preferences of the person who happens to do them, resulting in all sorts of technologies, protocols, formats, patterns, etc; REST, SOAP, SSIS, FTP, XML, JSON, ODBC, CSV, EDI. This also means that the integrations become highly person-dependent. The organization is now deep into the Emerging maturity level:
As long as there were few integrations, they were a mere nuisance, but as the need for integration increases the problems become apparent:
- The deceptively thin lines that often illustrate integrations are in reality small applications themselves, with their own bugs and need for updates, monitoring, etc, but they seldom get the proper attention.
- Creating an integration between two systems is like adding a reinforcing bar between them; the systems become dependent on each other (tight coupling) and they loose flexibility; certain changes in one system requires changes in the integrated systems too.
- Adding an integration means that someone temporarily understands both systems enough to make them communicate with each other. This knowledge is soon forgotten and the integration becomes a nightmare to maintain.
- The more integrations that a system has, the harder it is to update the system.
- Replacing a system that has many integrations is often compared to a heart transplantation and can easily bring the business development to a stand-still for a year.
- As a result, the IT landscape becomes less and less flexible over time, so changes are becoming more and more costly and the IT department becomes a bottle neck for business development-
- The business has no idea about what the integrations do and why - and this is also true for most people in the IT department.
- When we need a new integration, even if we have made a similar one before, we will still need a new one for the new need.
Centralizing the integration efforts
There have been serious attempts to centralize the integration, introduce integration tools and create central integration teams that are responsible for all the integrations. This solves just a few of the issues and it also adds new challenges:
- The integration team has to understand all the systems on a detailed level, which can be very challenging in organizations with many systems.
- The integration tools often requires special education and personnel with that knowledge is scarce and expensive.
- The central integration team can easily become a bottle neck for all projects that need new or updated integrations.
Web application integration
With the growth of web applications for customers, the need is no longer to exchange information between systems, but rather access the data and functionality of the systems. This introduces a new challenge which traditional integration platforms don't support very well: the systems need to expose their functionality to the web applications.
The response from the IT industry to these requirements was to provide proprietary API:s for each system. This is a clear indication that the organization is about to enter the Connected maturity level. This of course lead to new challenges such as:
- We are exposing sensitive data to Internet
- By opening up the systems to the customers, the systems get a much heavier load than they have been designed for
- The customers expect 24/7 accessibility
- It can be really hard to add an API to an existing system
- Proprietary API:s also means that you have to understand the details of each and every one of the API:s that you need as a web developer
- This adds a new tight coupling between the client (web application) and the server (the IT system)
- The applications become even less flexible; a change of their API will affect all web applications that are using it
If the organization tries to achieve the Connected maturity level without changing their integration architecture, they set themselves up for failure. The organization will end up with an extensive technology debt and a rigid IT landscape that threatens to slow the business development down to a grinding halt.
Digitalization of the business processes
With digital transformation, organizations no longer use IT only as a supporting tool for their business processes (in the form of vertical applications), they require the IT systems to transform into an integrated business platform that can run the processes end to end with maximum use of digital technology. This corresponds to the maturity levels Orchestrated and Ominiscient.
The fact that the IT industry has spent years trying to solve this and that solutions are still of ad hoc nature, adding more challenges than they solve, indicates that we need a major change in our approach. As organizations are embracing the digitalization wave, the need for integration increases dramatically. This will further expose the anomalies of the current integration paradigms and opens the door for an architecture shift.