The management and interpretation of data remains a challenge and one of many solutions is the use of data integration patterns to standardise the integration process.
First imagine if you can, the data stored on your phone. Then consider that data being collected by the apps and websites you use on your device: date, time, location, time on page, what you may have browsed or purchased and more. The sheer volume of data is mind boggling and that meagre list barely scratches the surface.
It’s easy to conclude that data is a complex and critical component to any business – a hugely valuable asset that provides intelligence enabling organisations to make better, more timely strategic decisions.
The management and interpretation of data remains a challenge and one of many solutions is the use of data integration patterns to standardise the integration process.
Like most patterns, creation comes from usage and can be optimised to meet the needs of the business. There are several different types of modern integration, each offering a unique solution that can be adopted based on the desired outcome.
Migration – flow of data in one direction
The most straightforward of patterns, and essential to all data systems, migration is simply movement from one system to another. You could be using a newer version of software, for example and want to move the data from one source to another.
Using specific criteria such as location of data to be migrated, the transformation the data will go through and the destination and results of migration, the data that has amassed over time is able to move seamlessly without risk of loss.
Not only is this an important pattern to prevent loss of historical data, but it presents a unified view of the data (e.g a dashboard of information about customers; the region, source, orders) and can make reporting much easier. This pattern of integration is most commonly used in larger organisations that have system updates in their roadmaps.
Aggregation – data from multiple systems into one aggregate
The key difference with aggregation of data is that it is smarter – it is primarily about assembling a unified view from multiple sources.
Consider any major freeway or motorway in the world, let’s take the SH 1 in New Zealand as an example: you have multiple flows and on-ramps coming from many different locations and the logic is to have them all in one place where you can see them at any given time.
You can decide what data you want to include, how frequently you run reports and which data is refreshed when you decide to run a report. For example, you may want to generate a report, but orders are more important to showcase than geo-location as it’s probable that people don’t move as often as they order. With aggregation, the data is also up to date when you need it and migration might be a once off or daily, enabling data to be pulled several times a day in real time, at any time.
The aggregation pattern is also incredibly useful for compliance or regulatory requirements as it ensures compliance data lives in one system, but may be an amalgamation of relevant data from multiple systems. Ultimately learning is reduced across systems, while you still have visibility of activity.
Bi-directional Sync/Flow – two systems that feed into each other
This is an exciting pattern that combines two datasets from two different systems into one, while acknowledging their need to exist as individual datasets. For example, you may harness two different powerhouse systems or tools that use the same data but for entirely different reasons and with different outcomes.
Here’s what it could look like: one system manages customer service and the other manages orders. To a certain extent, both use the same data and by using a bi-directional sync you can continue to use both systems (instead of one suite with a shared database), with real-time view of the data.
This enables a much clearer view of the customer based on the needs of the user – the manager of the warehouse using the system for users wants to see the customer name and address; whereas the customer service advisor would probably want the whole picture and customer history. Bi-directional synchronisation allows both parties to have a view of the customer through the lens they need.
Broadcast – covers event driven systems
Think of this as the social media of patterns: from one to many – which refers to moving data from a single source to multiple systems on an ongoing and real-time basis. The broadcast integration pattern really does enable a source of truth and event-driven modus operandi, with certain systems able to subscribe to notifications from other systems to obtain a piece of data.
Unlike the bi-directional integration pattern, broadcast only moves data in one direction, from the source to the destination and unlike the migration pattern, is transactional.
The broadcast pattern adds value when one system needs to recruit information from another system in real time. This can be applied in many ways with numerous ways in which this might work – the fulfilment of orders via your eCommerce platform, for example.
Correlation – updating data when needed
Say for example you work as a mechanic and you operate numerous walk-in garages across a city. You may have two sets of information stored at separate garages about the same customer – they could have visited at different periods in their life, with some of the data remaining the same (e.g their name). The correlation integration pattern identifies the intersection of two data sets and synchronises.
Correlation works in a similar way to bi-directional sync pattern, and is useful when you have two groups or systems that have data that represents the same item or person. In the case of the garages, the most recent visit to have a repair might enable the data to be updated and shared across the system where other data is held – a new address or new car. This will then provide an up to date record of the most recent action taken for this customer.
There is a beneficial value exchange here, as you only keep and store the data you need, which reduces the impact on storage cost and performance. This is a smart pattern that prioritises the data you are chiefly concerned with – scoping out unnecessary data.
Modern integration provides numerous operational and financial benefits for business and can streamline systems to perform more effectively over time.
Want to learn more? We have significant experience in modern integration and modern solution architecture. Our architecture principles always look to modern best practices with regards to designing in-line with DevOps/CD and loose coupling of components.
We are technology agnostic and always ensure the right technology or pattern is selected for the right purpose. This includes cloud architecture and infrastructure design across multiple cloud providers. Contact us today to set up a discovery call to go through your needs.
