Market Disruption with Real Time Data Insights

Technology is changing our world at a rapid rate. For companies to stay competitive they must adapt to the changing market conditions. This is particularly true for Data Analytics which is now moving into, what many call, the third wave – Real Time Data Insights.

Initially the storing and processing of data was compartmentalised either by department or by different product groups. Operational data was also used to provide data insights with a departmental view.

Customer orientation changed this in the 1990ies, with the need to understand and serve customers better, organisations started to build their analytics models around customer segments. This was supported by the rise of Data Warehouse systems that allowed a centralised view of the data across departments and products. Operational data was replicated, cleansed and enhanced to allow different views on customer segments and other relevant business views. The problems arising from this model was a costly transformation process coupled with the difficulty to integrate insights into operational processes and systems.

In recent years with the ubiquity of the smartphone revolution customers have become inherently more closely connected to organisations and expect personalised service for a more meaningful customer experience. This puts new challenges on organisations to identify the customer, rapidly classify them based on their behavioural patterns and turn these insights into a better, more personalised service or buying experiences.

The Third Wave

The other remarkable trend is that cloud computing provides a more cost-effective model to tailor analytics platforms utilising vast amounts of computing power and storage at reasonable prices. This allows companies to integrate analytics outputs in real time with operational systems. This improves and automates many customer interactions and provides better information to business staff.

This model also allows organisations to integrate machine learning and AI capabilities. Whilst not entirely new concepts, cloud computing has enabled organisations to use intelligent processes that automate the interaction with customers as well as improve the analytics capabilities.

Foundation to Success – Real Time Analytics Architecture

When building a new capability, the first step is to lay the foundation in form of an architecture that will not only allow traditional batch analytics, but also provide a speed layer that can quickly transform millions of transactions into insights that can help frontline staff  better serve the customer and personalise the customer experience.

Figure 1: Data Lake Context Diagram

At the core of the modern analytics platform sits an auto scalable processing capability in the form of clustered compute power. The most commonly used technology is Hadoop based clusters running a variety of different applications such as; Spark, Storm or Presto. The clusters can run in long running instances or as transient clusters that only live as long as the analytics process is running and are shut down immediately afterwards.

Modern cloud concepts such as infrastructure as code make it possible to provide flexible Continuous Integration(CI)/Continuous Deployment(CD) processes that stand up a new environment of hardware and software automatically within minutes. Other operational concepts allow Dev Ops teams to manage the data pipelines including reconciliation capability that is paramount when running integrated workloads.

The second important cornerstone of a successful analytics architecture is cheap blob storage with the ability to store large amounts of raw data at low cost. This storage is used as a staging area as well as for long term storage of raw data.

New data access and ETL tools allow direct analysis of the raw data. This enables Data Scientists to find new data patterns directly in the raw data. Working with raw data is a major improvement to the Extract, Transform and Load(ETL) processes in traditional Data Warehouses as it allows for unknown patterns to be retained in the data that would otherwise be lost.

To utilise the full potential of Realtime Data Analytics event driven access is required. This allows downstream systems to be notified when business insights and/or business events (represented as changes in data) are occurring that require timely responses. The service layer also requires an access model that can be provided either in form of APIs or other data access implementations.

Realtime Analytics will increasingly be an integral part of modern Enterprise Data Models enabling organisations to provide personalised services that will disrupt traditional market models. Progressive organisations should move into this space quickly with some low risk projects to understand the potential and complexities of modern analytics platforms. Disrupt or be disrupted!

Let Fusion Professionals work with you to help move your organisation towards a “Real Time” data insights landscape.

Fusion Insights

In recent years data volumes have been increasing dramatically. This has created major challenges for traditional analytics platforms in terms…

MORE INFORMATION

With the increasing volumes of data that can be cost effectively stored in the cloud, comes increasing responsibility. The current…

MORE INFORMATION

With the advancement of technology and abundance of data your business receives on a daily basis, companies are now in…

MORE INFORMATION

Fusion Professionals held its annual Fusion Summit last Thursday the 18th of October at the Rag and Famish Hotel in…

MORE INFORMATION

The Client is one of major NSW government departments providing services to public. The Department had been experiencing performance issues…

MORE INFORMATION

Though its conception dates back to 1979, containers made their mark as much needed, major technology assets in 2000. Digital…

MORE INFORMATION

Objective The intelligent mobile app-based lending system is a new field, blending recent technical developments in mobile phones and Artificial…

MORE INFORMATION

Our Client is a well-known Australian freight logistics company, operating in railway freight and shipping.  The company embarked on a…

MORE INFORMATION

Data warehouse management and data analytics always had the challenge to decide what data to store and for how long…

MORE INFORMATION

Cloud computing is becoming a preferred storage platform for IT managers and organisations in general. In Australia alone, 31 percent…

MORE INFORMATION

Serving your customer in the best possible, most efficient way should always be the major goal of any organisation. The…

MORE INFORMATION

Moving out from proprietary software seems like a daredevil act, considering the possible data security issues some open source databases…

MORE INFORMATION

The Challenge Complex IT environments can pose significant technical risk that, if not managed adequately, have the potential of major…

MORE INFORMATION

Fusion Professionals has signed a partnership agreement with Waterline Data ( https://www.waterlinedata.com/ ) the leading provider of Information Catalogs and…

MORE INFORMATION

Most people do not like change. As much as possible, they want things to stay the same that is why,…

MORE INFORMATION

Regardless of your infrastructure whether you are running in the cloud or on-premise, there will always be a need to…

MORE INFORMATION

Data Analytics tools provide Data Scientists and Data Engineers with the instruments to find patterns in data and provide business…

MORE INFORMATION

Fusion Professionals held its Fusion Meld 2018 event last Thursday, the 17th of May at the Terrace Hotel in North…

MORE INFORMATION

Over the past 10 years, data has grown into a behemoth that dominates business intelligence. A huge percentage of the…

MORE INFORMATION

A data lab is a well-equipped environment that allows organisations to explore and examine new ideas by combining new data…

MORE INFORMATION