Author: adminwcuk1

What are IBM Cloud Paks?

Beyond containers and Kubernetes, you need to orchestrate your production topology and provide management, security and governance for your applications. IBM Cloud Paks are enterprise-ready, containerized software solutions that run on Red Hat® OpenShift® on IBM Cloud™ and Red Hat Enterprise Linux. Built on a common integration layer, IBM Cloud Paks include containerized IBM middleware and common software services for development and management.

• IBM Cloud Pak™ for Applications. Quickly build cloud-native apps by leveraging built-in developer tools and processes, including support for microservices functions and serverless computing. 
• IBM Cloud Pak™ for Data. Simplify the collection, organization, and analysis of data. Turn data into insights through an integrated catalog of IBM, open source, and third-party microservices add-ons. 
• IBM Cloud Pak™ for Integration. Achieve the speed, flexibility, security, and scale required for all of your integration and digital transformation initiatives, including API lifecycle, application and data integration, messaging and events, high-speed transfer, and integration security
• IBM Cloud Pak™ for Automation. Deploy on your choice of clouds, with low-code tools for business users and real-time performance visibility for business managers. Migrate your automation runtimes without application changes or data migration. Automate at scale without vendor lock-in.
• IBM Cloud Pak™ for Multicloud Management. Gain consistent visibility, automation, and governance across a wide range of hybrid, multicloud management capabilities including integration with existing tools and processes
• IBM Cloud Pak™ for Security. Integrate security tools to gain insights into threats across hybrid, multicloud environments.

IBM Cloud Pak for Integration 

Building integrated solutions requires you to use more than one integration pattern at a time. Simplify the management of your integration architecture and reduce cost. Running on Red Hat OpenShift, IBM Cloud Pak for Integration gives you the agility to deploy workloads on-premises and on private and public clouds. 

• API lifecycle management. Create, secure, manage, share, and monetize APIs across clouds while you maintain continuous availability. 
• Application and data integration. Integrate your business data and applications quickly and easily across any cloud system
• Enterprise messaging. Simplify, accelerate, and facilitate the reliable exchange of data with a trusted, flexible, and security-rich messaging solution. 
• Event streaming. Use Apache Kafka to deliver messages more easily and reliably and to react to events in real time. 
• High-speed data transfer. Reliably send, share, stream, and sync large files and data sets at maximum speed. 
• Platform-level security, automation, and monitoring. Quickly set up and manage gateways, control access on a per resource basis, deploy your integration flows, and monitor all of your traffic.

API lifecycle management 

Bridge the gap between cloud and on-premises applications quickly and easily by abstracting your back-end implementation as APIs. One of the best ways to do this is by exposing services as APIs for external consumption and let the consuming applications compose the integration logic

• Expand. Provide a standard API interface. Include global API discovery to access key business functions as fine-grained services. Encourage data reuse and mashups driven by innovative transformation use cases.
• Integrate. Create a significant impact on your business goals by exposing core services through managed APIs. Enable projects to integrate with one another and discover the benefits of synergy across the enterprise.
• Scale. Be prepared to scale dynamically based on the demands of your expanding ecosystem and other usage metrics.

Application & data integration

Integrate all your business data and applications across any cloud more quickly and easily using open standards. From the simplest SaaS application to the most complex legacy systems, this pattern alleviates the concern about mismatched sources, formats, or standards. 

• Integrate applications. Connect applications and data sources on-premises or across multiple clouds to coordinate the exchange of business information as a coarse-grained service so that core data and transactions maintain their integrity. In contrast to the API management pattern, this pattern is best suited for coarse-grained services. 
• Integrate data. In near real time, synchronize data across multiple endpoints in the integration landscape to achieve a cohesive view of data, gathered from legacy back ends to SaaS applications, to DBaaS repositories, to analytics cloud services.
• Incorporate agile integration. Unify cross-enterprise capabilities. Enforce the use of core enterprise services and business processes. Include cognitive augmentation within your integration logic. Set up agile organizational models and governance practices.

Enterprise messaging

Simplify, accelerate, and facilitate the reliable exchange of data with a flexible and enhance security-rich IT services messaging solution. Extend traditional messaging capabilities in modern applications to communicate with new technologies from AI, IoT devices, and other digital channels. 

• Ensure secure and reliable messaging. Preserve message integrity throughout the network, protect data, and ensure regulatory compliance with security-rich functions. Provide reliable delivery without message loss, duplication, or complex recovery.
• Unify your enterprise. More easily integrate heterogeneous application platforms using industry-standard JMS messaging protocols, scalable publish-subscribe, and a choice of APIs
• Expect high performance and scalable message transfer. Your apps can rely on a highly available solution with fully automated failover, dynamically distributed messaging workloads, high throughput, and a low-latency solution.
• Simplify management and control. Use a dashboard to gain insights with visibility to message and file tracking. Audit data movement and transaction completion.

Event streaming

Take advantage of event streams to build adaptive solutions with engaging, more personalized user experiences by responding to events before the moment passes. By design, events occur in a continuous stream from a multitude of sources in a low-latency, high-velocity manner. 

• Decrease system complexity. Loose coupling allows event producers to emit events without any knowledge about who is going to consume those events. Likewise, event consumers don’t need to be aware of the event emitters. 
• Simplify the interface. One event producer can reach multiple endpoints with a single call.
• React to events as they happen. Enable the following scenarios: IoT device, streaming analytics, real-time back-end transactions, geolocation tracking, and auditing
• Facilitate machine learning. Improve predictive analytics by moving to real-time event streaming from batch processing

High-speed file transfer

Enterprises need a reliable, fast, secure data transfer and synchronized system that is hybrid and multi-cloud IT services. An integration platform makes it possible to securely transfer data across geographies faster than traditional tools, between any kind of storage, whether it’s on-premises, in the cloud, or across diverse cloud vendors.

• Integrate application data. Coordinate the exchange of business information so that data is available when and where it is needed.
• Transform data for analytics. Access, cleanse, and prepare data to create a consistent view of your business within a data warehouse or data lake.
• Enrich enterprise data. Augment DBaaS content with data from enterprise back-end systems for a 360-degree view of the user. Allow partner and in-house data sources to sync and complement each other’s updates
• Transfer data. Move huge amounts of data between on-premises and cloud or from cloud to cloud rapidly and predictably with enhanced levels of IT service security. Speed the adoption of cloud platforms when data is very large and needs to be exchanged across long distances. 

IBM Garage: Accelerate your journey

Modernization comes in many flavors, and rewriting your entire estate is not feasible. Big bang modernization efforts are risky, so it is best to break large initiatives into smaller projects with measurable impact. Your goal is to accelerate value, deliver frequently, and reduce risk. IBM Garage experts can help.

• Co-create. Identify a business modernization opportunity. Define and build the MVP with your squad, get feedback, and co-create a solution.
• Co-execute. Manage risk by choosing the right approach to modernize your current estate. Accelerate your journey through automation and technology
• Co-operate. Harden for production, standardize operations, and improve DevOps efficiency across your application estate. 

Integration Has Changed

IDC estimates that spending on digital transformation initiatives will represent a $20 trillion market opportunity over the next 5 years. What’s behind this staggering explosion of spending? The ever-present, ever-growing need to build new customer experiences through connected experiences across a network of applications that leverage data of all types

That’s no easy task – bringing together processes and information sources at the right time and in the right context is difficult at best, particularly when you consider the aggressive adoption of SaaS business applications. New data sources need to be injected into business processes to create competitive differentiation

The Value of Application Integration for Digital Transformation

When you consider your agenda for building new customer experiences and focus on how data is accessed and made available for the services and APIs that power these initiatives, you can see several significant benefits that application integration brings to the table:

• Effectively addressing disparity – Access data from any system in any format and build homogeneity from it, no matter how diverse your multicloud landscape grows
• Expertise of the endpoints – Modern integration includes smarts around complex protocols and data formats, but it also incorporates intelligence about the actual objects, business and functions within the end systems
• Innovation through data – Applications owe much of their innovation to their opportunity to combine data beyond their boundaries and create meaning from it, a trait particularly visible in microservices architecture
• Enterprise-grade artifacts – Integration flows inherit a tremendous amount of value from the runtime, which includes enterprise-grade features for error recovery, fault tolerance, log capture, performance analysis, and much more.

The integration landscape is changing to keep up with enterprise and marketplace computing demands, but how did we get from SOA and ESBs to a modern, containerized, agile approach to integration?

The Journey So Far – SOA and the ESB pattern

Before we can look forward to the future of agile integration, we need to understand what came before. SOA (service oriented architecture) patterns emerged at the start of the millennium, and at first the wide acceptance of the standards SOA was built upon heralded a bright future where every system could discover and talk to any other system via synchronous exposure patterns.

This was typically implemented in the form of the ESB (enterprise service bus) – an architectural pattern that was aimed at providing synchronous connectivity to backend systems typically over web or on-site embedded services. While many enterprises successfully implemented the ESB pattern, it became something of a victim of its own success.

• ESB patterns often formed a single infrastructure for the whole enterprise, with tens or hundreds of integrations installed on a production server cluster. Although heavy centralization isn’t required by the ESB pattern, the implemented topologies almost always fell prey to it. 
• Centralized ESB patterns often failed to deliver the significant savings companies were hoping for. Few interfaces could be re-used from one project to another, yet the creation and maintenance of interfaces was prohibitively expensive for any one project to take on. 
• SOA was more complex than just the implementation of an ESB, particularly around who would fund an enterprise-wide program. Cross Enterprise initiatives like SOA and its underlying ESB struggled to find funding, and often that funding only applied to services that would be reusable enough to cover their creation cost.

The result was that creation of services by this specialist SOA team sometimes became a bottleneck for projects rather than the enabler that it was intended to be. This typically gave the centralized ESB pattern a bad name by association.

All that said, the centralized ESB pattern does bring some benefits, especially if they have a highly skilled integration team with a low attrition rate, and who receive a predictable and manageable number of new integration requirements. A single, centralized ESB certainly simplifies consistency and governance of implementation. However, many organizations have more fluid and dynamic requirements to manage, and are also under pressure to implement integration using similar cloud native technologies and agile methods as are being used in other parts of the organization. A case in point is the move to microservices architecture typically found in the application development space. 

Service oriented architecture (SOA) vs microservice architecture

SOA and microservices architecture share many words in common, but they are in fact completely separate concepts.

Service-oriented architecture and the associated ESB pattern is an enterprise-wide initiative to make the data and functions in systems of record readily available to new applications. We create re-usable, synchronous interfaces such as web services and RESTful APIs to expose the systems of record, such that new innovative applications can be created more quickly by incorporating data from multiple systems in real time

Microservices architecture, on the other hand, is a way of writing an individual application as a set of smaller (microservice) components in a way that makes that application more agile, scalable, and resilient. So in summary, service oriented architecture is about real-time integration between applications, whereas microservices architecture is about how we build the internals of applications themselves. 

The Case for Agile Integration

Why have microservices concepts become so popular in the application space? They represent an alternative approach to structuring applications. Rather than an application being a large silo of code running on the same server, the application is designed as a collection of smaller, completely independently-running components.

Microservices architecture enables three critical benefits:

• Greater agility – Microservices are small enough to be understood completely in isolation and changed independently. 
• Elastic scalability – Their resource usage can be fully tied into the business model.
• Discrete resilience – With suitable decoupling, changes to one microservice do not affect others at runtime.

With those benefits in mind, what would it look like if we re-imagined integration, which is typically deployed in centralized silos, with a new perspective based on microservices architecture? That’s what we call an “Agile Integration.”

There are three related, but separate aspects to agile integration:

Aspect 1: Fine-grained integration deployment. What might we gain by breaking out the integrations in the siloed ESB into separate runtimes that could be maintained and scaled independently? What is the simplest way that these discrete integrations be made

Aspect 2: Decentralized integration ownership. How should we adjust the organizational structure to better leverage a more autonomous approach, giving application teams more control over the creation and exposure of their own integrations?

Aspect 3: Cloud native integration infrastructure. How can we best leverage the container-based infrastructure that underpins cloud native applications, to provides productivity, operational consistency, and portability for both applications and integrations across a hybrid and multi-cloud landscape

Aspect 1: Fine-grained Integration Deployment

Traditional integration is characterized by the heavily centralized deployment of integrations in the ESB pattern. Here, all integrations are deployed to a single heavily nurtured (HA) pair of integration servers has been shown to introduce a bottleneck for projects. Any deployment to the shared servers runs the risk of destabilizing existing critical interfaces. No individual project can choose to upgrade the version of the integration middleware to gain access to new features. 

Using the same concepts as microservice architecture, we could break up the enterprise-wide ESB into smaller, more manageable and dedicated pieces. Perhaps in some cases we can even get down to one runtime for each interface we expose. These “fine-grained integration deployment” patterns provide specialized, right-sized containers, offering improved agility, scalability and resilience, and look very different to the centralized ESB patterns of the past. Figure 1 demonstrates in simple terms how a centralized ESB differs from fine-grained integration deployment.

Fine-grained integration deployment draws on the benefits of a microservices architecture. Let’s revisit what we listed as microservices benefits in light of fine-grained integration deployment:

• Agility: Different teams can work on integrations independently without deferring to a centralized group or infrastructure that can quickly become a bottleneck. Individual integration flows can be changed, rebuilt, and deployed independently of other flows, enabling safer application of changes and maximizing speed to production.
• Scalability: Individual flows can be scaled on their own, allowing you to take advantage of efficient elastic scaling of cloud infrastructures.
• Resilience: Isolated integration flows that are deployed in separate containers cannot affect one another by stealing shared resources, such as memory, connections, or CPU.

Aspect 2: Decentralized integration ownership

A significant challenge faced by service-oriented architecture was the way that it tended to force the creation of centralized integration teams and infrastructure to implement the service layer. 

This created ongoing friction in the pace at which projects could run since they always had the central integration team as a dependency. The central team knew their integration technology well, but often didn’t understand the applications they were integrating, so translating requirements could be slow and error prone. 

Many organizations would have preferred the application teams own the creation of their own services, but the technology and infrastructure of the time didn’t enable that. 

The move to fine-grained integration deployment opens a door such that ownership of the creation and maintenance of integrations can also be distributed out to the application teams. It’s not unreasonable for business application teams to take on integration work, streamlining the implementation of new integrations. 

Furthermore, API management has matured to the point where application teams can easily manage the exposure of their own APIs, again without resorting to a separate centralized integration team. 

Microservices design patterns often prefer to increase decoupling by receiving event streams of data and building localized data representations rather than always going via API calls to retrieve data in real time. Agile integration also considers how best to enable teams to publish and consume event streams both within and beyond application boundaries.

Aspect 3: Cloud-native integration infrastructure

Integration runtimes have changed dramatically in recent years. So much so that these lightweight runtimes can be used in truly cloud-native ways. By this we are referring to their ability to hand off the burden of many of their previously proprietary mechanisms for cluster management, scaling, availability and to the cloud platform in which they are running.

This entails a lot more than just running them in a containerized environment. It means they have to be able to function as “cattle not pets,” making best use of the orchestration capabilities such as Kubernetes and many other common cloud standard frameworks. 

Clearly, Agile Integration requires that the integration topology be deployed very differently. A key aspect of that is a modern integration runtime that can be run in a container-based environment and is well suited to cloudnative deployment techniques. Modern integration runtimes are almost unrecognizable from their historical peers. Let’s have a look at some of those differences:

• Fast lightweight runtime: They run in containers such as Docker and are sufficiently lightweight that they can be started and stopped in seconds and can be easily administered by orchestration frameworks such as Kubernetes.
• Dependency free: They no longer require databases or message queues, although obviously, they are very adept at connecting to them if they need to. 
• File system based installation: They can be installed simply by laying their binaries out on a file system and starting them up—ideal for the layered file systems of Docker images. 
• DevOps tooling support: The runtime should be continuous integration and deployment-ready. Script and property file-based install, build, deploy, and configuration to enable “infrastructure as code” practices. Template scripts for standard build and deploy tools should be provided to accelerate inclusion into DevOps pipelines.
• API-first: The primary communication protocol should be RESTful APIs. Exposing integrations as RESTful APIs should be trivial and based upon common conventions such as the Open API specification. Calling downstream RESTful APIs should be equally trivial, including discovery via definition files.
• Digital connectivity: In addition to the rich enterprise connectivity that has always been provided by integration runtimes, they must also connect to modern resources. For example, NoSQL databases (MongoDb and Cloudant etc.), and messaging services such as Kafka. Furthermore, they need access to a rich catalogue of application intelligent connectors for SaaS (software as a service) applications such as Salesforce
• Continuous delivery: Continuous delivery is enabled by command line interfaces and template scripts that mesh into standard DevOps pipeline tools. This further reduces the knowledge required to implement interfaces and increases the pace of delivery
• Enhanced tooling: Enhanced tooling for integration means most interfaces can be built by configuration alone, often by individuals with no integration background. With the addition of templates for common integration patterns, integration best practices are burned into the tooling, further simplifying the tasks. Deep integration specialists are less often required, and some integration can potentially be taken on by application teams as we will see in the next section on decentralized integration. 

Modern integration runtimes are well suited to the three aspects of an agile integration methodology: fine-grained deployment, decentralized ownership, and true cloud-native infrastructure. 

Along with integration runtimes becoming more lightweight and container friendly, we also see API management and messaging/eventing infrastructure moving to container-based deployment. This is generally in order to benefit from the operational constancy provided by orchestration platforms such as Kubernetes that provides auto scaling, load-balancing, deployment, internal routing, reinstatement and more in a standardized way, significantly simplifying the administration of the platform.

Agile Integration for the Integration Platform

Throughout this paper, we have been focused on the application integration features as deployed in an agile integration architecture. However, many enterprise solutions can only be solved by applying several critical integration capabilities. An integration platform (or what some analysts refer to as a “hybrid integration platform”) brings together these capabilities so that organizations can build business solutions in a more efficient and consistent way. 

Many industry specialists agree on the value of this integration platform. Gartner notes:

The hybrid integration platform (HIP) is a framework of on-premises and cloud-based integration and governance capabilities that enables differently skilled personas (integration specialists and non-specialists) to support a wide range of integration use cases.… Application leaders responsible for integration should leverage the HIP capabilities framework to modernize their integration strategies and infrastructure, so they can address the emerging use cases for digital business. 

One of the key things that Gartner notes is that the integration platform allows multiple people from across the organization to work in user experiences that best fits their needs. This means that business users can be productive in a simpler experience that guides them through solving straightforward problems, while TeraPixels Systems IT specialists in San Diego have expert levels of control to deal with the more complex enterprise scenarios. These users can then work together through reuse of the assets that have been shared; while preserving governance across the whole.

Satisfying the emerging use cases of the digital transformation is as important as supporting the various user communities. The bulk of this paper will explore these emerging use cases, but first we should further elaborate on the key capabilities that must be part of the integration platform.

IBM Cloud Pak for Integration

IBM Cloud Integration brings together the key set of integration capabilities into a coherent platform that is simple, fast and trusted. It allows you to easily build powerful integrations and APIs in minutes, provides leading performance and scalability, and offers unmatched end-to-end capabilities with enterprise-grade security. IBM Cloud Pak for Integration is built on the open source Kubernetes platform for container orchestration. 

IBM Cloud Pak for Integration is the most complete hybrid integration platform in the industry including all of the key integration capabilities your team needs:

Application and Data Integration Connects applications and data sources on-premises or in the cloud, in order to coordinate exchange business information so that data is available when and where needed.

API Lifecycle Exposes and manages business services as reusable APIs for select developer communities both internal and external to your organization. Organizations adopt an API strategy to accelerate how effectively they can share their unique data and services assets to then fuel new applications and new business opportunities.

Enterprise Messaging Ensures real-time information is available from anywhere at anytime by providing reliable message delivery without message loss, duplication or complex recovery in the event of a system or network issue. 

High Speed Data Transfer: Move huge amounts of data between on-premises and cloud or cloud-to cloud rapidly and predictably with enhanced levels of security. Facilitates how quickly organizations can adopt cloud platforms when data is very large

Secure Gateway Extend Connectivity and Integration beyond the enterprise with DMZ-ready edge capabilities that protect APIs, the data they move, and the systems behind them

Integration is the lifeblood of today’s digital economy. Hybrid integration is a key business imperative for most enterprises, as digitalization has led to a proliferation of applications, services, APIs, and data stores that need to be connected to realize end-to-end functionality and, in many cases, an entirely new digital business proposition. A hybrid integration platform caters to a range of integration needs, including on-premises app integration, cloud application integration, messaging, event streaming, rapid API creation and lifecycle management, B2B/EDI integration, mobile application/back-end integration, and file transfer. User productivity tools and deployment flexibility are key characteristics of a hybrid integration platform that helps enterprises respond faster to evolving digital business requirements. 

Ovum view

Ovum ICT Enterprise Insights 2018 survey results indicate a strong inclination on the part of IT leaders to invest in integration infrastructure modernization, including the adoption of new integration platforms. IT services professionals in Orange County and other areas continue to struggle to meet new application and data integration requirements driven by digitalization and changing customer expectations. Line-of-business (LoB) leaders are no longer willing to wait for months for the delivery of integration capabilities that are mission-critical for specific business initiatives. Furthermore, integration competency centers (ICCs) or integration centers of excellence (COEs) are being pushed hard to look for alternatives that significantly reduce time to value without prolonged procurement cycles.

Digital business calls for flexible integration capabilities that connect diverse applications, services, APIs, and data stores; hybrid integration continues to be a complex IT issue. The current enterprise IT agenda gives top priority to connecting an ever-increasing number of endpoints and mitigating islands of IT infrastructure and information silos that make the vision of a “connected enterprise” difficult to achieve. Hybrid integration, which involves disparate applications, data formats, deployment models, and transactions, is a multifaceted problem for which there is no simple solution. For example, while an enterprise service bus (ESB) can be appropriate for data/protocol transformation and on-premises application integration, integration PaaS (iPaaS) is clearly a popular solution for SaaS-to-on-premises and SaaS-to-SaaS integration.

The center-of-gravity hypothesis applies to integration architecture. There is a greater inclination to deploy integration platforms closer to applications and data sources. APIs continue to gain prominence as flexible interfaces to digital business services and enablers for enterprises looking to innovate and participate in the wider digital economy. The unrelenting drive toward SaaS is leading to a rapid shift of integration processes to the cloud. A combination of these trends is driving the emergence of a new agile hybrid integration paradigm, with cloud-based integration platforms used for cloud, mobile application/back-end, B2B/EDI, and data integration. This integration paradigm or pattern is gaining popularity as enterprises do not have the luxury of executing dedicated, cost intensive and time-consuming integration projects to meet digitalization-led, hybrid integration requirements. Enterprise IT leaders realize that existing legacy integration infrastructure offers less flexibility and is difficult to maintain, so they are now more open to new integration approaches or platforms that improve developer productivity and allow them to “do more with less.” Moreover, traditional, heavyweight middleware is a barrier for enterprises looking to achieve agile hybrid integration to meet critical digital business requirements.

Agile hybrid integration calls for modular solutions that integrate well with each other and offer a uniform user experience (UX) and developer productivity tools to reduce time to integration and cost of ownership. For example, enterprises need to achieve integration within a few days of subscribing to a new set of SaaS applications, and frequently need to expose SaaS applications via representational state transfer (REST) APIs for consumption by mobile applications. They may also need to design and manage a new set of APIs for externalization of the enterprise or monetization of new applications and enterprise data assets. A hybrid integration platform can meet all these requirements, with modular integration solutions deployed on-premises, in the cloud, or on software containers according to the requirements of specific use cases. 

In the background of changing digital business requirements, IT leaders need to focus on revamping their enterprise integration strategy, which invariably will involve adoption of a hybrid integration platform that offers deployment and operational flexibility and greater agility at a lower cost of ownership to meet multifaceted hybrid integration requirements. Integration modernization initiatives aim to use new integration patterns, development and cultural practices, and flexible deployment options to drive business agility and reduce costs. It is important to identify a strategic partner (and not just a software vendor with systems integration capabilities) that can provide essential advice and best practices based on years of practical experience to ensure that integration modernization initiatives stay on track and deliver desired outcomes. 

Recommendations

• Enterprise IT leaders should focus on developing a forward-looking strategy for hybrid integration using the best of existing on-premises middleware and specific cloud-based integration services (i.e., PaaS products for hybrid integration). For all practical purposes, and in most cases, it would make sense to opt for a hybrid integration platform. This does not imply a complete “rip and replace” strategy for deciding the future of existing on-premises middleware. With DevOps practices, microservices, and containerized applications gaining popularity, IT leaders should evaluate the option of deploying middleware (integration platforms) on software containers as a means to driving operational agility and deployment flexibility
• With several middleware vendors focusing on developing a substantial proposition for hybrid integration, it would be better to exploit a more cohesive set of integration capabilities provided by the same vendor. A “do it yourself” approach to integration or federation between middleware products offered by different vendors is rarely easy, and it is of course easier to train users on a hybrid integration platform offering a uniform UX
• Integration is still predominantly carried out by IT service practitioners; however, IT leaders should consider “ease of use” for both integration practitioners and less skilled, non-technical users (e.g., power users) when selecting integration platforms for a range of hybrid integration use 

Integration modernization is a recurring theme driven by digitalization and the need for greater agility 

Hybrid integration complexity continues to drive integration modernization

Over the last couple of years, “integration modernization” has regularly featured in Ovum’s conversations with enterprise IT leaders. Digitalization has led to an almost unrelenting need for expose and consume APIs and exploiting digital assets to cater for ever-changing customer requirements and drive growth via new digital business models. Digital business initiatives call for more open, agile, and API-led integration capabilities, reducing time to integration. Enterprises need to develop customer-centric and more flexible business processes that can easily be extended via APIs to a range of access channels. The business side is asking some tough questions, including how fast and at what cost IT can deliver the desired integration capabilities. Ovum ICT Enterprise Insights 2018 survey results show that over 60% of respondent enterprises are planning substantial investment (including strategic investment in new iPaaS solutions) in iPaaS solutions over the next 18-month period. The survey results indicate that about 58% of respondent enterprises are planning substantial investment in API platforms over the same period. These figures clearly indicate enterprise interest in investing in new integration platforms to tackle hybrid integration challenges

Hybrid integration platform

Hybrid integration involves a mix of on-premises, cloud, B2B/EDI, mobile application/back-end integration, rapid API creation and lifecycle management, messaging, events, and file transfer use case scenarios of varying complexity (see Figure 1). Owing to specific business-IT requirements, enterprises may not have the flexibility to use “on-premises only” middleware or only cloud-based integration platforms. In certain cases, even the same integration capabilities (e.g., API management) need to be used both as on-premises middleware and as a cloud service (i.e., PaaS).  

An important aspect of hybrid integration requirements driven by digitalization is the need to support a range of user personas, including application developers, integration practitioners, enterprise/solution architects, and less skilled business users (i.e., non-technical users). Given the persistent time and budget constraints, enterprises often do not have the luxury of deploying only technical resources for hybrid integration initiatives and ICCs/integration COEs are not always in the driver’s seat. Simplified and uniform UX, self-service integration capabilities, and developer productivity tools are therefore critical in meeting hybrid integration requirements.

Ovum defines a hybrid integration platform as a cohesive set of integration software (middleware) products that enable users to develop, secure, and govern integration flows, connecting diverse applications, systems, services, and data stores, as well as enabling rapid API creation/composition and lifecycle management to meet the requirements of a range of hybrid integration use cases. A hybrid integration platform is “deployment model agnostic” in terms of delivering requisite integration capabilities, be it on-premises and cloud deployments or containerized middleware.

The key characteristics of a hybrid integration platform include:

• support for a range of application, service, and data integration use cases, with an API-led, agile approach to integration, reducing development effort and costs 
• uniformity in UX across different integration products or use cases and for a specific user persona 
• uniformity in underlying infrastructure resources and enabling technologies 
• flexible integration at a product or component API level 
• self-service capabilities for enabling less skilled, non-technical users 
• the flexibility to rapidly provision various combinations of cloud-based integration services based on specific requirements 
• openness to federation with external, traditional on-premises middleware platforms 
• support for embedding integration capabilities (via APIs) into a range of applications or solutions
• developer productivity tools (e.g., a “drag-and-drop” approach to integration flow development and pre-built connectors and templates) and their extension to a broader set of integration capabilities 
• flexible deployment options: on-premises deployment, public, private, and hybrid cloud deployment, and containerization 
• centralization of administration and governance capabilities.

Specific features and capabilities of hybrid integration platforms vary from vendor to vendor, and certain hybrid integration platforms may not offer some of the above-specified capabilities. It is noteworthy that the evolution from traditional middleware and PaaS for specific integration use cases (e.g., iPaaS for SaaS integration) to a hybrid integration platform is a work in progress for a majority of middleware vendors. 

iPaaS is now a default option for SaaS integration, and the iPaaS model for delivery of cloud integration capabilities is no longer about only offering dozens or hundreds of connectors and pre-built integration templates. It is important for iPaaS vendors to target new user personas and a broader set of integration use cases. In this context, we see two key developments: self-service integration capabilities for less skilled, non-technical user enablement, and artificial intelligence (AI)/machine learning (ML) capabilities simplifying development of integration flows

Terapixels offers IT services in San Diego that has a hybrid of on site and remote monitoring. In addition, this hybrid environments call for a cloud-native integration paradigm that readily supports DevOps practices and drives operational agility by reducing the burden associated with cluster management, scaling, and availability. As per such a cloud-native integration paradigm, integration runtimes run on software containers, are continuous integration and continuous delivery and deployment (CI/CD) ready, and are significantly lightweight and responsive enough to start and stop within a few seconds. Many enterprises have made substantial progress in containerizing applications to benefit from a microservices architecture and portability across public, private, and hybrid cloud

microservices architecture and portability across public, private, and hybrid cloud environments. Containerized applications and middleware represent a good combination; in cases where an application and a runtime are packaged and deployed together, developers can benefit from container portability and ease of use offered by the application and middleware combination. 

It also makes sense for applications and middleware to share a common architecture, as DevOps teams can then avoid the overhead and complexity associated with the proposition of running containerized applications on different hardware and following different processes to the existing ones with traditional middleware. This is true even in cases that do not involve much re-architecting of the applications; DevOps teams can still develop and deploy faster using fewer resources

Developers are increasingly building APIs that support new applications that use loosely coupled microservices. Each microservice has a particular function that can be independently scaled or maintained without impacting other loosely coupled services. A microservices architecture can involve both internal and external APIs, with internal APIs invoked for inter-service communication and external API calls initiated by API consumers. IT leaders must realize that microservices management is different in scope from API management and focus on effectively meeting both requirements

Good API design and operations principles (i.e., API- and design-first principles) are gaining ground in enterprises that have previous experience of experimenting with enterprise API initiatives linked to new digital business services. Consequently, API platforms are gaining traction. Multicloud API management and deployment on software containers are areas of significant interest to large enterprises. An API platform enables users to develop, run, manage, and secure APIs and microservices, and offers a superset of capabilities in comparison to those provided by API lifecycle management solutions. As the graphical approach to integration flows provided by application integration capabilities can now be deployed as microservices, these technologies jointly provide a holistic approach to the rapid creation/composition of APIs and the subsequent management of their lifecycle and operations. A key benefit of an API platform is the ability to create, test, and implement an API rapidly and reiterate the cycle to create a new version of it based on user feedback (i.e., the application of DevOps-style techniques to API lifecycle and operations).  

Internet of Things (IoT) integration use cases call for message-oriented middleware (MoM) that offer standards-based message queue (MQ) middleware to ease integration with enterprise applications and data stores. It is particularly suitable for heterogeneous environments, as any type of data can be transported as messages; MQ middleware is frequently used in mainframe, cloud, mobile, and IoT integration use case scenarios. A hybrid integration platform should support integration requirements of such use cases.

A lot of data is generated in the form of streams of events, with publishers creating events and subscribers consuming these events in different ways or via different means. Event-driven applications can deliver better customer experiences. For example, this could be in the form of adding context to ML models to obtain real-time recommendations that evolve continually as per the requirements of a specific use case. Embedding real-time intelligence into applications and real-time reaction or responsiveness to events are key capabilities in this regard.

For distributed applications using microservices, developers can opt for asynchronous event-driven integration, in addition to the use of synchronous integration and APIs. Apache Kafka, an open source stream-processing platform, is a good option for such use cases that require high throughput and scalability. Kubernetes can be used as a scalable platform for hosting Apache Kafka applications. As Apache Kafka reduces the need for point-to-point integration for data sharing, it can reduce latency to just a few milliseconds, thereby enabling faster delivery of data to the users. A hybrid integration platform should cater to the integration requirements of event-driven applications.

A hybrid integration platform with simplified UX, scalable architecture, and flexible deployment options

Key attributes at architectural and operational levels simplify hybrid integration and drive developer productivity and cost savings

The IBM Cloud Pak for Integration (shown in Figure 2) solves a range of hybrid integration requirements, including on-premises and SaaS application and data integration, rapid API creation/composition and lifecycle management, API security and API monetization, messaging, event streaming, and high-speed transfer. IBM offers a holistic integration platform exploiting a container based portable architecture for a range of hybrid integration use cases, as well as providing essential advice and support to help enterprises succeed with their integration modernization initiatives.

IBM Cloud Pak for Integration was built for deployment on containers and provides a modern architecture that includes the management of containerized applications and Kubernetes, an open source container orchestration system. An interesting trend is the adoption of DevOps culture, microservices, and PaaS for responsiveness to changes driven by digital business requirements. With IBM Cloud Pak for Integration’s container-based architecture, users have the flexibility to deploy on any environment that has Kubernetes infrastructure, as well as exploit a self-service approach to integration. IBM Cloud Pak for Integration enables simplified creation and reuse of integrations, their deployment close to the source, and self-service integration to deliver faster time to integration at lower cost. It offers the benefit of a unified UX for developing and sharing integrations, which promotes integration asset reuse to improve developer productivity.

With IBM Cloud Pak for Integration, users can deploy integration capabilities easily onto a Kubernetes environment. This provision helps achieve faster time to value for integration modernization initiatives by integrating the monitoring, logging, and IT security systems of a private cloud environment to ensure uniformity across a cloud integration platform deployment. Containerization fosters the flexibility of cloud private architecture, thereby helping users meet performance and scalability requirements as specified in the service-level agreements (SLAs) of their business applications. Another benefit is common administration and governance enabled via a single point of accessibility. This mitigates the need for logging in to multiple tools and better supports access management across different teams. In terms of deployment flexibility, IBM supports deployment on any cloud or on-premises deployment.

IBM espouses an approach that differentiates API management from microservices management but also combines the two to offer more than the sum of the parts. Istio running on Kubernetes allows users to manage the interactions between microservices running in containers. Integration between Security Gateway and Istio service mesh (involving security, application resiliency, and dynamic routing between microservices) can offer a good solution to end-to-end routing. IBM has optimized the gateway for cloud-native workloads. An interesting trend is the growth in the number of API providers offering additional endpoints to adapt to emerging architectural styles, such as GraphQL. GraphQL APIs have the ability to use a single query to fetch required data from multiple resources. IBM is extending its integration platform’s API capabilities to provide support for GraphQL management, and this approach decouples GraphQL management from GraphQL server implementation. 

IBM’s Agile Integration methodology

Agile integration focuses on delivering business agility as part of an integration modernization initiative. It espouses the transition of integration ownership from centralized integration teams to application teams, as supported by the operational consistency achieved via containerization. On the operational agility side, cloud-native infrastructure offers dynamic scalability and resilience. 

A good case in point is a fine-grained integration deployment pattern involving specialized, right-sized containers that deliver improved agility, scalability, and resilience. This is quite different from traditional, centralized ESB patterns, which is why IBM redesigned each of these capabilities, including the application integration features, to be deployed in a microservices-aligned manner. With a fine-grained deployment pattern, enterprises can improve build independence and production speed to drive deployment agility. In a nutshell, as part of integration modernization initiatives, “agile integration” caters to people, processes, and technology aspects to provide necessary advice and guidance to help enterprises achieve faster time to value across diverse deployment environments.