Case Study : Big Data solution for real time customer insights at leading Airline
Large Australian Airline with over 30,000 employees and more than 6000 daily flights.
In the Airline industry booking data is the essential driver for many business decisions and is an important input in many operational and sales processes. To provide the best service to customers the Airline requires this data and associated customer insights in near real time to be provided to the greater business community.
The objective was to build a platform that can provide data from various different operational data sources, analyse this data in near real time and provide the information to business systems.
Some data structures contain more than 3000 fields that are updated over 1 million times per day with peaks of over 80 transactions per second.
The solution was built in line with a typical Bigdata architecture. It is based on the Hadoop Ecosystem hosted on the AWS cloud platform Elastic Map Reduce and a number of peripheral systems allowing ingestion and event driven distribution of data to business systems in near real time.
The platform ingests data from a variety of data sources. The main source is the Amadeus Booking system that provides a live feed of booking updates in a continuously ingested stream. Other real time data feeds include Flight updates and Customer Profile information. In the future more Data Sources will be added with the aim to enrich the data and to provide the ability to perform cross domain analytics.
Processing is based on Clustered computing models using the Apache Hadoop ecosystem. Spark Streaming is used for long running, streaming applications that process up to 500 transactions per second. Transient clusters use the capabilities of code driven cloud deployments to generate the infrastructure on the fly and shutdown once the analytics job is complete. This allows cost effective and flexible processing for complex analytical jobs with large amounts of complex data.
Data is stored predominantly in clustered relational database systems and simple storage solutions that allow structured as well as unstructured data to be stored.
Data is served to operational and business systems that require access to real time customer insights. The Data API exposes the data via an API Gateway to internal system or third party partners.
An Event API provides the means to notify interested systems of changes in the underlying data. This way systems can make full use of the real-time nature of the data and the analytical capability.
Fusion Professionals provided technical leadership to make this project a successful implementation. Our staff provided Solution Architecture, Development Management, established development processes and standards as well as implemented large parts of the framework.
BEST FOR BIG DATA
Fusion Professionals is very experienced in providing streaming Bigdata platforms based on Hadoop and Spark frameworks. Our technical ability and architectural excellence helps customers move to the modern, integrated analytics platforms for 21st century business models.
The delivered solution is a platform rather than a standalone system. It is an extendable data analytics platform that will allow the airline to process large amounts of data, process machine learning experiments as well as do the heavy lifting in complex analytics processes that interact directly with operational systems.
Here are some of the highlights:
- Near real time analytics capability
- Bigdata processing platform
- Architecture framework
- Event driven API interface to socialise real time customer insights
- Fully cloud hosted
- Continuous deployment of infrastructure and applications
- Operational architecture for monitoring and management
Cloud hosted platforms can provide extreme processing power at relatively low costs. This can be achieved with on demand deployment and automatic scaling of environments. Spot pricing can be used to create large processing clusters at a fraction of the cost of inhouse analytics platforms.
Information cycles are becoming faster with customers expecting personalised service experiences. This is only possible if data is analysed fast and made available to the business units in real time:
- Real time customer insights
- Better service ability through faster data cycles
- Ability to engage with customers in a more personalised customer experience
- Improved profitability through better customer sentiment
Whilst this sort of business model is very cutting edge today in only a few years it will be a must for all companies to know their customers and their daily interactions with the company in real time.
This bigdata platform is designed to perform complex processing in extremely short time frames. Here are some of the benchmarks achieved during testing and in production:
- Over 500 transactions per second
- 15-20 second latency from business transaction to insights delivered
- Average processing time within the system 2-3 seconds
- Over 900 API calls per second (testing limit)
- High Availability with inbuilt Disaster Recovery
- Auto Scaling and Auto Healing
- Deployment of complete environment in under 50 minutes
Start your journey into the 21st century data processing today and let us talk to you about your requirements to move your business closer to your customers.
Moving out from proprietary software seems like a daredevil act, considering the possible data security issues some open source databases…MORE INFORMATION
The Challenge Complex IT environments can pose significant technical risk that, if not managed adequately, have the potential of major…MORE INFORMATION
Fusion Professionals has signed a partnership agreement with Waterline Data ( https://www.waterlinedata.com/ ) the leading provider of Information Catalogs and…MORE INFORMATION
Most people do not like change. As much as possible, they want things to stay the same that is why,…MORE INFORMATION
Regardless of your infrastructure whether you are running in the cloud or on-premise, there will always be a need to…MORE INFORMATION
Data Analytics tools provide Data Scientists and Data Engineers with the instruments to find patterns in data and provide business…MORE INFORMATION
Fusion Professionals held its Fusion Meld 2018 event last Thursday, the 17th of May at the Terrace Hotel in North…MORE INFORMATION
Over the past 10 years, data has grown into a behemoth that dominates business intelligence. A huge percentage of the…MORE INFORMATION
A data lab is a well-equipped environment that allows organisations to explore and examine new ideas by combining new data…MORE INFORMATION
Technology is changing our world at a rapid rate. For companies to stay competitive they must adapt to the changing…MORE INFORMATION
Another 12 months passes and the AWS Summit rolls into town, now hosted at the new impressive Sydney ICC! So what…MORE INFORMATION
DevOps has well and truly arrived. Having a team combining development and operations is superseding the traditional model where these…MORE INFORMATION
We may not know it, but we’re consuming huge amounts of data every day. Whether it’s through Siri, Google, Microsoft,…MORE INFORMATION
Fusion Professionals, a Sydney-based IT consulting firm and a Gold level member of the Oracle Partner Network (OPN) today announced…MORE INFORMATION
This week Fusion Professionals held their annual summit at the Rag & Famish hotel in North Sydney. The summit is…MORE INFORMATION
Working with a large, well-resourced enterprise has many benefits – including having the scope to look at new ways of…MORE INFORMATION
Who doesn’t like some certainty in their lives?! It’s human nature to crave it. So, when we are lining up…MORE INFORMATION
Building a business case is a familiar and routine process for any IT investment project. But is there a possibility…MORE INFORMATION
Fusion Professionals and Australian research and advisory firm BigInsights, recently hosted an industry breakfast to discuss how organisations can capitalise…MORE INFORMATION
Are you considering taking advantage of the Amazon Web Services (AWS) platform but are concerned about the risks involved in…MORE INFORMATION