Market Disruption with Real Time Data Insights

Technology is changing our world at a rapid rate. For companies to stay competitive they must adapt to the changing market conditions. This is particularly true for Data Analytics which is now moving into, what many call, the third wave – Real Time Data Insights.

Initially the storing and processing of data was compartmentalised either by department or by different product groups. Operational data was also used to provide data insights with a departmental view.

Customer orientation changed this in the 1990ies, with the need to understand and serve customers better, organisations started to build their analytics models around customer segments. This was supported by the rise of Data Warehouse systems that allowed a centralised view of the data across departments and products. Operational data was replicated, cleansed and enhanced to allow different views on customer segments and other relevant business views. The problems arising from this model was a costly transformation process coupled with the difficulty to integrate insights into operational processes and systems.

In recent years with the ubiquity of the smartphone revolution customers have become inherently more closely connected to organisations and expect personalised service for a more meaningful customer experience. This puts new challenges on organisations to identify the customer, rapidly classify them based on their behavioural patterns and turn these insights into a better, more personalised service or buying experiences.

The Third Wave

The other remarkable trend is that cloud computing provides a more cost-effective model to tailor analytics platforms utilising vast amounts of computing power and storage at reasonable prices. This allows companies to integrate analytics outputs in real time with operational systems. This improves and automates many customer interactions and provides better information to business staff.

This model also allows organisations to integrate machine learning and AI capabilities. Whilst not entirely new concepts, cloud computing has enabled organisations to use intelligent processes that automate the interaction with customers as well as improve the analytics capabilities.

Foundation to Success – Real Time Analytics Architecture

When building a new capability, the first step is to lay the foundation in form of an architecture that will not only allow traditional batch analytics, but also provide a speed layer that can quickly transform millions of transactions into insights that can help frontline staff  better serve the customer and personalise the customer experience.

Figure 1: Data Lake Context Diagram

At the core of the modern analytics platform sits an auto scalable processing capability in the form of clustered compute power. The most commonly used technology is Hadoop based clusters running a variety of different applications such as; Spark, Storm or Presto. The clusters can run in long running instances or as transient clusters that only live as long as the analytics process is running and are shut down immediately afterwards.

Modern cloud concepts such as infrastructure as code make it possible to provide flexible Continuous Integration(CI)/Continuous Deployment(CD) processes that stand up a new environment of hardware and software automatically within minutes. Other operational concepts allow Dev Ops teams to manage the data pipelines including reconciliation capability that is paramount when running integrated workloads.

The second important cornerstone of a successful analytics architecture is cheap blob storage with the ability to store large amounts of raw data at low cost. This storage is used as a staging area as well as for long term storage of raw data.

New data access and ETL tools allow direct analysis of the raw data. This enables Data Scientists to find new data patterns directly in the raw data. Working with raw data is a major improvement to the Extract, Transform and Load(ETL) processes in traditional Data Warehouses as it allows for unknown patterns to be retained in the data that would otherwise be lost.

To utilise the full potential of Realtime Data Analytics event driven access is required. This allows downstream systems to be notified when business insights and/or business events (represented as changes in data) are occurring that require timely responses. The service layer also requires an access model that can be provided either in form of APIs or other data access implementations.

Realtime Analytics will increasingly be an integral part of modern Enterprise Data Models enabling organisations to provide personalised services that will disrupt traditional market models. Progressive organisations should move into this space quickly with some low risk projects to understand the potential and complexities of modern analytics platforms. Disrupt or be disrupted!

Let Fusion Professionals work with you to help move your organisation towards a “Real Time” data insights landscape.

Fusion Insights

Fusion Professionals held its Fusion Meld 2018 event last Thursday, the 17th of May at the Terrace Hotel in North…

MORE INFORMATION

Over the past 10 years, data has grown into a behemoth that dominates business intelligence. A huge percentage of the…

MORE INFORMATION

A data lab is a well-equipped environment that allows organisations to explore and examine new ideas by combining new data…

MORE INFORMATION

Technology is changing our world at a rapid rate. For companies to stay competitive they must adapt to the changing…

MORE INFORMATION

Another 12 months passes and the AWS Summit rolls into town, now hosted at the new impressive Sydney ICC!   So what…

MORE INFORMATION

DevOps has well and truly arrived. Having a team combining development and operations is superseding the traditional model where these…

MORE INFORMATION

We may not know it, but we’re consuming huge amounts of data every day. Whether it’s through Siri, Google, Microsoft,…

MORE INFORMATION

Fusion Professionals, a Sydney-based IT consulting firm and a Gold level member of the Oracle Partner Network (OPN) today announced…

MORE INFORMATION

This week Fusion Professionals held their annual summit at the Rag & Famish hotel in North Sydney. The summit is…

MORE INFORMATION

Working with a large, well-resourced enterprise has many benefits –  including having the scope to look at new ways of…

MORE INFORMATION

Who doesn’t like some certainty in their lives?! It’s human nature to crave it. So, when we are lining up…

MORE INFORMATION

Building a business case is a familiar and routine process for any IT investment project. But is there a possibility…

MORE INFORMATION

Fusion Professionals and Australian research and advisory firm BigInsights, recently hosted an industry breakfast to discuss how organisations can capitalise…

MORE INFORMATION

Are you considering taking advantage of the Amazon Web Services (AWS) platform but are concerned about the risks involved in…

MORE INFORMATION

Fusion Professionals is a proud sponsor of the Nikola Tesla – Unlimited Mind Exhibition in Sydney.  For the very first…

MORE INFORMATION

Large Australian Airline with over 30,000 employees and more than 6000 daily flights. OBJECTIVES In the Airline industry booking data…

MORE INFORMATION

It’s been a while since Oracle held a business event in Sydney but the recent Oracle Modern Business Experience (MBX)…

MORE INFORMATION

The Client is a Government electricity distributor, providing the safe and reliable supply of electricity to 2.4 million people in…

MORE INFORMATION

The “WannaCry” ransomware campaign has targeted a number of organisations internationally including the UK’s National Health Service and Spanish telecommunications…

MORE INFORMATION

The Client is an Australian freight logistics company, operating in railway freight and shipping. The company embarked on a major…

MORE INFORMATION