Data Governance – Nuisance or Opportunity
With the increasing volumes of data that can be cost effectively stored in the cloud, comes increasing responsibility. The current trend in the industry shows increasing regulatory requirements for; privacy, storing data, managing the lineage of data and securing the data sufficiently.
Whilst Data Governance is not a new concept it is increasingly more relevant in modern enterprises. Especially, since the more stringent requirements of the European, General Data Protection Regulation (GDPR) have taken effect. Companies have found themselves needing to review and update their processes and security frameworks.
In many cases, this has caused organisations to struggle with compliance and many moan about the effort and cost that it takes to implement the new processes and safeguards. Many regard the new regulations as too stringent and see the implementation as a nuisance that should be kept to the bare minimum.
However, others have turned a problem into an opportunity. As we are entering the third wave of business insights, data storage and analytics are moving into more de-centralised silos that are closer to production and transactional systems. This data is only loosely coupled and is left mostly in its original format allowing a reduction of complex and expensive ETL processes.
This has multiple advantages. Business insights can be surfaced much faster and directly integrated into the business process. The other main advantage is that instead of moving data into a centralised Data Warehouse it can be processed at the point of acquisition. This directly provides data driven benefits to customers and frontline business units, reducing the data cycle to seconds and minutes rather than hours and days.
The result is that businesses can create better services and improved sales opportunities. Many organisations have used this to disrupt established markets to the point that companies following traditional business models struggle to keep up.
Decentralisation of data responsibility into business domains requires a Governance Model that allows decentralised data management and centralised oversight.
Cornerstone of this model is a well-designed architecture that creates the foundation of how data is stored, what tools are available and how overall security is implemented and managed.
When we talk about decentralised data responsibility we do not mean that the data is necessarily stored in different locations. Cloud computing makes data increasingly location agnostic. What we mean is that instead of having a centralised BI team that owns all the data and business insights, individual business domains take responsibility of the data. This is achieved with Data Stewards assigned to the business domains.
Data Stewards are responsible for their data domain, maintaining the inventory of data types and elements. Updating a centralised data catalogue that not only contains all meta data for individual data domains, but that also pro-actively suggests connections between domains is a key to a successful governance framework.
Data Triage is becoming an important part of managing enterprise wide data connections. This is the responsibility of a Data Council, consisting of the Data Stewards from all domains, to determine and govern enterprise wide policies, standards and processes especially for data that goes across boundaries.
How does this help the organisation? It addresses a number of issues that companies face with the massive increase of data volumes in recent years. Firstly, they can gradually move away from expensive monolithic data warehouse solutions and achieve a best of breed architecture that keeps them current in the fast changing technology landscape of modern analytics platforms.
Secondly, they resolve many issues with domain oriented analytics solutions built by business units that currently sit in spreadsheets and MS Access tables. Whilst this might not remove these it helps to better integrate these into a wider framework or better yet, remove the need as a faster time to market can be achieved with Domain ownership of data.
Finally, it provides a more comprehensive data management governance structure, as the ownership of data is decentralised with a centralised Governance model that allows the building of well designed controls into the organisation without putting up too much red tape or creating delivery bottlenecks in a centralised team.
Let Fusion Professionals work with you to develop the most appropriate Data Governance framework and strategy for your organisation.
Achim Drescher is the Managing Consultant of the Big Data and Analytics Practice at Fusion Professionals.
With 30 years in the IT industry, he is an Expert in Enterprise Software and Data Architecture, Data Governance frameworks and modern analytics platforms for Big Data and Data Lakes.
Statistical language models apply probability distributions to a sequence of words. These models are finding increasing use as natural language…MORE INFORMATION
Challenges The Company, one of Australia’s largest and fastest growing Telco companies had 2 primary SharePoint environments that had different…MORE INFORMATION
Containerization allows applications to run on any machine- anytime, anywhere so long as they are compatible. By virtualizing your OS,…MORE INFORMATION
So you’ve finally decided that the cloud is safer than corporate data centers and digital assets and you’ve chosen to…MORE INFORMATION
Building a system that houses your organisation’s data can be daunting, especially now that data acquisition is growing rapidly. The…MORE INFORMATION
Human-to-machine communication has not yet been perfected, but enterprises are already beginning to integrate this groundbreaking technology into their operations,…MORE INFORMATION
Fusion Professionals has signed a partnership agreement with MapR Technologies, provider of the industry’s leading data platform for AI and…MORE INFORMATION
“Big data is at the foundation of all of the megatrends that are happening today, from social to mobile to…MORE INFORMATION
In recent years data volumes have been increasing dramatically. This has created major challenges for traditional analytics platforms in terms…MORE INFORMATION
With the increasing volumes of data that can be cost effectively stored in the cloud, comes increasing responsibility. The current…MORE INFORMATION
With the advancement of technology and abundance of data your business receives on a daily basis, companies are now in…MORE INFORMATION
Fusion Professionals held its annual Fusion Summit last Thursday the 18th of October at the Rag and Famish Hotel in…MORE INFORMATION
The Client is one of major NSW government departments providing services to public. The Department had been experiencing performance issues…MORE INFORMATION
Though its conception dates back to 1979, containers made their mark as much needed, major technology assets in 2000. Digital…MORE INFORMATION
Objective The intelligent mobile app-based lending system is a new field, blending recent technical developments in mobile phones and Artificial…MORE INFORMATION
Our Client is a well-known Australian freight logistics company, operating in railway freight and shipping. The company embarked on a…MORE INFORMATION
Data warehouse management and data analytics always had the challenge to decide what data to store and for how long…MORE INFORMATION
Cloud computing is becoming a preferred storage platform for IT managers and organisations in general. In Australia alone, 31 percent…MORE INFORMATION
Serving your customer in the best possible, most efficient way should always be the major goal of any organisation. The…MORE INFORMATION
Moving out from proprietary software seems like a daredevil act, considering the possible data security issues some open source databases…MORE INFORMATION