Case Study : Big Data solution for real time customer insights at leading Airline
Large Australian Airline with over 30,000 employees and more than 6000 daily flights.
In the Airline industry booking data is the essential driver for many business decisions and is an important input in many operational and sales processes. To provide the best service to customers the Airline requires this data and associated customer insights in near real time to be provided to the greater business community.
The objective was to build a platform that can provide data from various different operational data sources, analyse this data in near real time and provide the information to business systems.
Some data structures contain more than 3000 fields that are updated over 1 million times per day with peaks of over 80 transactions per second.
The solution was built in line with a typical Bigdata architecture. It is based on the Hadoop Ecosystem hosted on the AWS cloud platform Elastic Map Reduce and a number of peripheral systems allowing ingestion and event driven distribution of data to business systems in near real time.
The platform ingests data from a variety of data sources. The main source is the Amadeus Booking system that provides a live feed of booking updates in a continuously ingested stream. Other real time data feeds include Flight updates and Customer Profile information. In the future more Data Sources will be added with the aim to enrich the data and to provide the ability to perform cross domain analytics.
Processing is based on Clustered computing models using the Apache Hadoop ecosystem. Spark Streaming is used for long running, streaming applications that process up to 500 transactions per second. Transient clusters use the capabilities of code driven cloud deployments to generate the infrastructure on the fly and shutdown once the analytics job is complete. This allows cost effective and flexible processing for complex analytical jobs with large amounts of complex data.
Data is stored predominantly in clustered relational database systems and simple storage solutions that allow structured as well as unstructured data to be stored.
Data is served to operational and business systems that require access to real time customer insights. The Data API exposes the data via an API Gateway to internal system or third party partners.
An Event API provides the means to notify interested systems of changes in the underlying data. This way systems can make full use of the real-time nature of the data and the analytical capability.
Fusion Professionals provided technical leadership to make this project a successful implementation. Our staff provided Solution Architecture, Development Management, established development processes and standards as well as implemented large parts of the framework.
BEST FOR BIG DATA
Fusion Professionals is very experienced in providing streaming Bigdata platforms based on Hadoop and Spark frameworks. Our technical ability and architectural excellence helps customers move to the modern, integrated analytics platforms for 21st century business models.
The delivered solution is a platform rather than a standalone system. It is an extendable data analytics platform that will allow the airline to process large amounts of data, process machine learning experiments as well as do the heavy lifting in complex analytics processes that interact directly with operational systems.
Here are some of the highlights:
- Near real time analytics capability
- Bigdata processing platform
- Architecture framework
- Event driven API interface to socialise real time customer insights
- Fully cloud hosted
- Continuous deployment of infrastructure and applications
- Operational architecture for monitoring and management
Cloud hosted platforms can provide extreme processing power at relatively low costs. This can be achieved with on demand deployment and automatic scaling of environments. Spot pricing can be used to create large processing clusters at a fraction of the cost of inhouse analytics platforms.
Information cycles are becoming faster with customers expecting personalised service experiences. This is only possible if data is analysed fast and made available to the business units in real time:
- Real time customer insights
- Better service ability through faster data cycles
- Ability to engage with customers in a more personalised customer experience
- Improved profitability through better customer sentiment
Whilst this sort of business model is very cutting edge today in only a few years it will be a must for all companies to know their customers and their daily interactions with the company in real time.
This bigdata platform is designed to perform complex processing in extremely short time frames. Here are some of the benchmarks achieved during testing and in production:
- Over 500 transactions per second
- 15-20 second latency from business transaction to insights delivered
- Average processing time within the system 2-3 seconds
- Over 900 API calls per second (testing limit)
- High Availability with inbuilt Disaster Recovery
- Auto Scaling and Auto Healing
- Deployment of complete environment in under 50 minutes
Start your journey into the 21st century data processing today and let us talk to you about your requirements to move your business closer to your customers.
Many organisations don’t realise it, but in our current environment Data has become the main differentiator in the market. Most…MORE INFORMATION
Professional services, one of the fastest growing sectors of the Australian economy, covers a broad group of companies and organizations…MORE INFORMATION
We experience an increasing polarisation in our political landscape with tribalism becoming a real issue. This is partially to be…MORE INFORMATION
Oracle’s introduction of the self-driving, self-securing, and self-repairing Autonomous Database draws upon its decades of expertise in automating databases and…MORE INFORMATION
In a recent blog post from Dataiku, the leading data science, machine learning, and AI platform, Lynn Heidmann explored ways…MORE INFORMATION
“With Great Power Comes Great Responsibility” One of the biggest ongoing responsibilities that comes after commissioning an Exadata appliance is…MORE INFORMATION
According to Constellation Research, a little more than half of traditional Fortune 500 companies have disappeared due to the lack…MORE INFORMATION
Fusion Professionals has signed a partnership agreement with Dataiku, one of the world’s leading machine learning platforms that moves companies…MORE INFORMATION
Statistical language models apply probability distributions to a sequence of words. These models are finding increasing use as natural language…MORE INFORMATION
Challenges The Company, one of Australia’s largest and fastest growing Telco companies had 2 primary SharePoint environments that had different…MORE INFORMATION
Containerization allows applications to run on any machine- anytime, anywhere so long as they are compatible. By virtualizing your OS,…MORE INFORMATION
So you’ve finally decided that the cloud is safer than corporate data centers and digital assets and you’ve chosen to…MORE INFORMATION
Building a system that houses your organisation’s data can be daunting, especially now that data acquisition is growing rapidly. The…MORE INFORMATION
Human-to-machine communication has not yet been perfected, but enterprises are already beginning to integrate this groundbreaking technology into their operations,…MORE INFORMATION
Fusion Professionals has signed a partnership agreement with MapR Technologies, provider of the industry’s leading data platform for AI and…MORE INFORMATION
“Big data is at the foundation of all of the megatrends that are happening today, from social to mobile to…MORE INFORMATION
In recent years data volumes have been increasing dramatically. This has created major challenges for traditional analytics platforms in terms…MORE INFORMATION
With the increasing volumes of data that can be cost effectively stored in the cloud, comes increasing responsibility. The current…MORE INFORMATION
With the advancement of technology and abundance of data your business receives on a daily basis, companies are now in…MORE INFORMATION