Hybrid Integration Platforms: Digital Business Calls for Integration Modernization and Greater Agility

Integration is the lifeblood of today’s digital economy. Hybrid integration is a key business imperative for most enterprises, as digitalization has led to a proliferation of applications, services, APIs, and data stores that need to be connected to realize end-to-end functionality and, in many cases, an entirely new digital business proposition. A hybrid integration platform caters to a range of integration needs, including on-premises app integration, cloud application integration, messaging, event streaming, rapid API creation and lifecycle management, B2B/EDI integration, mobile application/back-end integration, and file transfer. User productivity tools and deployment flexibility are key characteristics of a hybrid integration platform that helps enterprises respond faster to evolving digital business requirements. 

Ovum view

Ovum ICT Enterprise Insights 2018 survey results indicate a strong inclination on the part of IT leaders to invest in integration infrastructure modernization, including the adoption of new integration platforms. IT services professionals in Orange County and other areas continue to struggle to meet new application and data integration requirements driven by digitalization and changing customer expectations. Line-of-business (LoB) leaders are no longer willing to wait for months for the delivery of integration capabilities that are mission-critical for specific business initiatives. Furthermore, integration competency centers (ICCs) or integration centers of excellence (COEs) are being pushed hard to look for alternatives that significantly reduce time to value without prolonged procurement cycles.

Digital business calls for flexible integration capabilities that connect diverse applications, services, APIs, and data stores; hybrid integration continues to be a complex IT issue. The current enterprise IT agenda gives top priority to connecting an ever-increasing number of endpoints and mitigating islands of IT infrastructure and information silos that make the vision of a “connected enterprise” difficult to achieve. Hybrid integration, which involves disparate applications, data formats, deployment models, and transactions, is a multifaceted problem for which there is no simple solution. For example, while an enterprise service bus (ESB) can be appropriate for data/protocol transformation and on-premises application integration, integration PaaS (iPaaS) is clearly a popular solution for SaaS-to-on-premises and SaaS-to-SaaS integration.

The center-of-gravity hypothesis applies to integration architecture. There is a greater inclination to deploy integration platforms closer to applications and data sources. APIs continue to gain prominence as flexible interfaces to digital business services and enablers for enterprises looking to innovate and participate in the wider digital economy. The unrelenting drive toward SaaS is leading to a rapid shift of integration processes to the cloud. A combination of these trends is driving the emergence of a new agile hybrid integration paradigm, with cloud-based integration platforms used for cloud, mobile application/back-end, B2B/EDI, and data integration. This integration paradigm or pattern is gaining popularity as enterprises do not have the luxury of executing dedicated, cost intensive and time-consuming integration projects to meet digitalization-led, hybrid integration requirements. Enterprise IT leaders realize that existing legacy integration infrastructure offers less flexibility and is difficult to maintain, so they are now more open to new integration approaches or platforms that improve developer productivity and allow them to “do more with less.” Moreover, traditional, heavyweight middleware is a barrier for enterprises looking to achieve agile hybrid integration to meet critical digital business requirements.

Agile hybrid integration calls for modular solutions that integrate well with each other and offer a uniform user experience (UX) and developer productivity tools to reduce time to integration and cost of ownership. For example, enterprises need to achieve integration within a few days of subscribing to a new set of SaaS applications, and frequently need to expose SaaS applications via representational state transfer (REST) APIs for consumption by mobile applications. They may also need to design and manage a new set of APIs for externalization of the enterprise or monetization of new applications and enterprise data assets. A hybrid integration platform can meet all these requirements, with modular integration solutions deployed on-premises, in the cloud, or on software containers according to the requirements of specific use cases. 

In the background of changing digital business requirements, IT leaders need to focus on revamping their enterprise integration strategy, which invariably will involve adoption of a hybrid integration platform that offers deployment and operational flexibility and greater agility at a lower cost of ownership to meet multifaceted hybrid integration requirements. Integration modernization initiatives aim to use new integration patterns, development and cultural practices, and flexible deployment options to drive business agility and reduce costs. It is important to identify a strategic partner (and not just a software vendor with systems integration capabilities) that can provide essential advice and best practices based on years of practical experience to ensure that integration modernization initiatives stay on track and deliver desired outcomes. 

Recommendations

• Enterprise IT leaders should focus on developing a forward-looking strategy for hybrid integration using the best of existing on-premises middleware and specific cloud-based integration services (i.e., PaaS products for hybrid integration). For all practical purposes, and in most cases, it would make sense to opt for a hybrid integration platform. This does not imply a complete “rip and replace” strategy for deciding the future of existing on-premises middleware. With DevOps practices, microservices, and containerized applications gaining popularity, IT leaders should evaluate the option of deploying middleware (integration platforms) on software containers as a means to driving operational agility and deployment flexibility
• With several middleware vendors focusing on developing a substantial proposition for hybrid integration, it would be better to exploit a more cohesive set of integration capabilities provided by the same vendor. A “do it yourself” approach to integration or federation between middleware products offered by different vendors is rarely easy, and it is of course easier to train users on a hybrid integration platform offering a uniform UX
• Integration is still predominantly carried out by IT service practitioners; however, IT leaders should consider “ease of use” for both integration practitioners and less skilled, non-technical users (e.g., power users) when selecting integration platforms for a range of hybrid integration use

Integration modernization is a recurring theme driven by digitalization and the need for greater agility

Hybrid integration complexity continues to drive integration modernization

Over the last couple of years, “integration modernization” has regularly featured in Ovum’s conversations with enterprise IT leaders. Digitalization has led to an almost unrelenting need for expose and consume APIs and exploiting digital assets to cater for ever-changing customer requirements and drive growth via new digital business models. Digital business initiatives call for more open, agile, and API-led integration capabilities, reducing time to integration. Enterprises need to develop customer-centric and more flexible business processes that can easily be extended via APIs to a range of access channels. The business side is asking some tough questions, including how fast and at what cost IT can deliver the desired integration capabilities. Ovum ICT Enterprise Insights 2018 survey results show that over 60% of respondent enterprises are planning substantial investment (including strategic investment in new iPaaS solutions) in iPaaS solutions over the next 18-month period. The survey results indicate that about 58% of respondent enterprises are planning substantial investment in API platforms over the same period. These figures clearly indicate enterprise interest in investing in new integration platforms to tackle hybrid integration challenges

Hybrid integration platform

Hybrid integration involves a mix of on-premises, cloud, B2B/EDI, mobile application/back-end integration, rapid API creation and lifecycle management, messaging, events, and file transfer use case scenarios of varying complexity (see Figure 1). Owing to specific business-IT requirements, enterprises may not have the flexibility to use “on-premises only” middleware or only cloud-based integration platforms. In certain cases, even the same integration capabilities (e.g., API management) need to be used both as on-premises middleware and as a cloud service (i.e., PaaS).  

An important aspect of hybrid integration requirements driven by digitalization is the need to support a range of user personas, including application developers, integration practitioners, enterprise/solution architects, and less skilled business users (i.e., non-technical users). Given the persistent time and budget constraints, enterprises often do not have the luxury of deploying only technical resources for hybrid integration initiatives and ICCs/integration COEs are not always in the driver’s seat. Simplified and uniform UX, self-service integration capabilities, and developer productivity tools are therefore critical in meeting hybrid integration requirements.

Ovum defines a hybrid integration platform as a cohesive set of integration software (middleware) products that enable users to develop, secure, and govern integration flows, connecting diverse applications, systems, services, and data stores, as well as enabling rapid API creation/composition and lifecycle management to meet the requirements of a range of hybrid integration use cases. A hybrid integration platform is “deployment model agnostic” in terms of delivering requisite integration capabilities, be it on-premises and cloud deployments or containerized middleware.

The key characteristics of a hybrid integration platform include:

• support for a range of application, service, and data integration use cases, with an API-led, agile approach to integration, reducing development effort and costs
• uniformity in UX across different integration products or use cases and for a specific user persona
• uniformity in underlying infrastructure resources and enabling technologies
• flexible integration at a product or component API level
• self-service capabilities for enabling less skilled, non-technical users
• the flexibility to rapidly provision various combinations of cloud-based integration services based on specific requirements
• openness to federation with external, traditional on-premises middleware platforms
• support for embedding integration capabilities (via APIs) into a range of applications or solutions
• developer productivity tools (e.g., a “drag-and-drop” approach to integration flow development and pre-built connectors and templates) and their extension to a broader set of integration capabilities
• flexible deployment options: on-premises deployment, public, private, and hybrid cloud deployment, and containerization
• centralization of administration and governance capabilities.

Specific features and capabilities of hybrid integration platforms vary from vendor to vendor, and certain hybrid integration platforms may not offer some of the above-specified capabilities. It is noteworthy that the evolution from traditional middleware and PaaS for specific integration use cases (e.g., iPaaS for SaaS integration) to a hybrid integration platform is a work in progress for a majority of middleware vendors. 

iPaaS is now a default option for SaaS integration, and the iPaaS model for delivery of cloud integration capabilities is no longer about only offering dozens or hundreds of connectors and pre-built integration templates. It is important for iPaaS vendors to target new user personas and a broader set of integration use cases. In this context, we see two key developments: self-service integration capabilities for less skilled, non-technical user enablement, and artificial intelligence (AI)/machine learning (ML) capabilities simplifying development of integration flows

Terapixels offers IT services in San Diego that has a hybrid of on site and remote monitoring. In addition, this hybrid environments call for a cloud-native integration paradigm that readily supports DevOps practices and drives operational agility by reducing the burden associated with cluster management, scaling, and availability. As per such a cloud-native integration paradigm, integration runtimes run on software containers, are continuous integration and continuous delivery and deployment (CI/CD) ready, and are significantly lightweight and responsive enough to start and stop within a few seconds. Many enterprises have made substantial progress in containerizing applications to benefit from a microservices architecture and portability across public, private, and hybrid cloud

microservices architecture and portability across public, private, and hybrid cloud environments. Containerized applications and middleware represent a good combination; in cases where an application and a runtime are packaged and deployed together, developers can benefit from container portability and ease of use offered by the application and middleware combination. 

It also makes sense for applications and middleware to share a common architecture, as DevOps teams can then avoid the overhead and complexity associated with the proposition of running containerized applications on different hardware and following different processes to the existing ones with traditional middleware. This is true even in cases that do not involve much re-architecting of the applications; DevOps teams can still develop and deploy faster using fewer resources

Developers are increasingly building APIs that support new applications that use loosely coupled microservices. Each microservice has a particular function that can be independently scaled or maintained without impacting other loosely coupled services. A microservices architecture can involve both internal and external APIs, with internal APIs invoked for inter-service communication and external API calls initiated by API consumers. IT leaders must realize that microservices management is different in scope from API management and focus on effectively meeting both requirements

Good API design and operations principles (i.e., API- and design-first principles) are gaining ground in enterprises that have previous experience of experimenting with enterprise API initiatives linked to new digital business services. Consequently, API platforms are gaining traction. Multicloud API management and deployment on software containers are areas of significant interest to large enterprises. An API platform enables users to develop, run, manage, and secure APIs and microservices, and offers a superset of capabilities in comparison to those provided by API lifecycle management solutions. As the graphical approach to integration flows provided by application integration capabilities can now be deployed as microservices, these technologies jointly provide a holistic approach to the rapid creation/composition of APIs and the subsequent management of their lifecycle and operations. A key benefit of an API platform is the ability to create, test, and implement an API rapidly and reiterate the cycle to create a new version of it based on user feedback (i.e., the application of DevOps-style techniques to API lifecycle and operations).  

Internet of Things (IoT) integration use cases call for message-oriented middleware (MoM) that offer standards-based message queue (MQ) middleware to ease integration with enterprise applications and data stores. It is particularly suitable for heterogeneous environments, as any type of data can be transported as messages; MQ middleware is frequently used in mainframe, cloud, mobile, and IoT integration use case scenarios. A hybrid integration platform should support integration requirements of such use cases.

A lot of data is generated in the form of streams of events, with publishers creating events and subscribers consuming these events in different ways or via different means. Event-driven applications can deliver better customer experiences. For example, this could be in the form of adding context to ML models to obtain real-time recommendations that evolve continually as per the requirements of a specific use case. Embedding real-time intelligence into applications and real-time reaction or responsiveness to events are key capabilities in this regard.

For distributed applications using microservices, developers can opt for asynchronous event-driven integration, in addition to the use of synchronous integration and APIs. Apache Kafka, an open source stream-processing platform, is a good option for such use cases that require high throughput and scalability. Kubernetes can be used as a scalable platform for hosting Apache Kafka applications. As Apache Kafka reduces the need for point-to-point integration for data sharing, it can reduce latency to just a few milliseconds, thereby enabling faster delivery of data to the users. A hybrid integration platform should cater to the integration requirements of event-driven applications.

A hybrid integration platform with simplified UX, scalable architecture, and flexible deployment options

Key attributes at architectural and operational levels simplify hybrid integration and drive developer productivity and cost savings

The IBM Cloud Pak for Integration (shown in Figure 2) solves a range of hybrid integration requirements, including on-premises and SaaS application and data integration, rapid API creation/composition and lifecycle management, API security and API monetization, messaging, event streaming, and high-speed transfer. IBM offers a holistic integration platform exploiting a container based portable architecture for a range of hybrid integration use cases, as well as providing essential advice and support to help enterprises succeed with their integration modernization initiatives.

IBM Cloud Pak for Integration was built for deployment on containers and provides a modern architecture that includes the management of containerized applications and Kubernetes, an open source container orchestration system. An interesting trend is the adoption of DevOps culture, microservices, and PaaS for responsiveness to changes driven by digital business requirements. With IBM Cloud Pak for Integration’s container-based architecture, users have the flexibility to deploy on any environment that has Kubernetes infrastructure, as well as exploit a self-service approach to integration. IBM Cloud Pak for Integration enables simplified creation and reuse of integrations, their deployment close to the source, and self-service integration to deliver faster time to integration at lower cost. It offers the benefit of a unified UX for developing and sharing integrations, which promotes integration asset reuse to improve developer productivity.

With IBM Cloud Pak for Integration, users can deploy integration capabilities easily onto a Kubernetes environment. This provision helps achieve faster time to value for integration modernization initiatives by integrating the monitoring, logging, and IT security systems of a private cloud environment to ensure uniformity across a cloud integration platform deployment. Containerization fosters the flexibility of cloud private architecture, thereby helping users meet performance and scalability requirements as specified in the service-level agreements (SLAs) of their business applications. Another benefit is common administration and governance enabled via a single point of accessibility. This mitigates the need for logging in to multiple tools and better supports access management across different teams. In terms of deployment flexibility, IBM supports deployment on any cloud or on-premises deployment.

IBM espouses an approach that differentiates API management from microservices management but also combines the two to offer more than the sum of the parts. Istio running on Kubernetes allows users to manage the interactions between microservices running in containers. Integration between Security Gateway and Istio service mesh (involving security, application resiliency, and dynamic routing between microservices) can offer a good solution to end-to-end routing. IBM has optimized the gateway for cloud-native workloads. An interesting trend is the growth in the number of API providers offering additional endpoints to adapt to emerging architectural styles, such as GraphQL. GraphQL APIs have the ability to use a single query to fetch required data from multiple resources. IBM is extending its integration platform’s API capabilities to provide support for GraphQL management, and this approach decouples GraphQL management from GraphQL server implementation. 

IBM’s Agile Integration methodology

Agile integration focuses on delivering business agility as part of an integration modernization initiative. It espouses the transition of integration ownership from centralized integration teams to application teams, as supported by the operational consistency achieved via containerization. On the operational agility side, cloud-native infrastructure offers dynamic scalability and resilience. 

A good case in point is a fine-grained integration deployment pattern involving specialized, right-sized containers that deliver improved agility, scalability, and resilience. This is quite different from traditional, centralized ESB patterns, which is why IBM redesigned each of these capabilities, including the application integration features, to be deployed in a microservices-aligned manner. With a fine-grained deployment pattern, enterprises can improve build independence and production speed to drive deployment agility. In a nutshell, as part of integration modernization initiatives, “agile integration” caters to people, processes, and technology aspects to provide necessary advice and guidance to help enterprises achieve faster time to value across diverse deployment environments.