What happened in Gartner Data & Analytics Summit 2022? What is Data Observability? And why is it important? What are the complexities of a Data Value Chain? What are the lessons learnt from building Self-Service Data Platform? What are the DataOps Observability Pyramid of Needs?
Let’s check out in the first newsletter…
Key Takeaways from Gartner Data & Analytics Summit 2022
Augmented analysis, synthetic data, adaptive governance, and more
Here are the four big ideas you should know from this year’s Summit:
- Change data from a liability to an asset with small data, synthetic data, and active metadata
- Treat analytics as both an art and a science with augmented analytics
- Become a “Decision Designer” to accelerate decisions and deliver insights at the right time
- Balance control and enablement with three types of governance
Data Quality Monitoring is dead. Say Hello to Full Data Stack Observability
Or how to unlock the reliability of your data assets at any stage of the pipeline
Data-driven decision-making is becoming instrumental in the growth of any modern organization.
But the modern data stack is evolving and getting more complex day by day.
Which has increased unpredictability and risk of failure to data pipelines.
Making compromises on data quality and data governance, however, is dangerous for organizations.
A new approach to dealing with data quality, trust, and reliability has emerged as data observability.
We need to ponder upon these questions:
- What is data observability?
- Data observability vs. testing and data quality monitoring — What’s the difference?
- Full Data Stack Observability — What does it mean?
- What are the most powerful applications of full data stack observability?
Complexity in a Data Value Chain
Complexity is tough for humans to handle.
In 2007 an HBR paper broke down decision-making into a model with four domains “chaotic — complex — complicated — and simple”.
While the whole paper on the Cynefin model is worthwhile reading, it had this, in particular, to say about complexity:
“the complex domain is much more prevalent in the business world than most leaders realize — and requires different, often counterintuitive, responses”
Cyenfin framework states that:
Leaders who try to impose order in a complex context will fail, but those who set the stage, step back a bit, allow patterns to emerge, and determine which ones are desirable will succeed.
In above context, “models” are useful to “set the stage” in order to allow the patterns to emerge.
But not all models are created equal.
And that’s where we can bring Wardley Maps back into the picture.
Finally, models are useful but finding the right model is more useful.
Source: https://medium.com/@nickj69/complexity-in-a-data-value-chain-281e0f266391
Lessons Learnt From Building Self-Service Data Platform
Is distributed data ownership right for your organization?
Challenges of Self-Service Data Platform
- Leadership's buy-in
- Lack of relevant skills
- Reinventing the wheel
- Managing access
Overcoming the Challenges
- Socialise
- Start small-scale
- Guardrails
- Improve collaboration
The DataOps Observability Pyramid of Needs
Is any data flowing?
Is data arriving in a usable window of time? (Are you meeting your SLAs?)
Is the data coming through valid and complete? Are there errors in the data itself?
Are there important changes in the data that I should know about?
How do issues mentioned, combined with organizational policies, map to problems in how people are actually working with the data?
Source: https://medium.com/databand-ai/dataops-observability-pyramid-of-needs-69b6a38bd6d
Subscribe to my Newsletter, Follow me on LinkedIn, and never miss updates again.
What do you think about my weekly Newsletter?
If you have any suggestions, want me to feature your article, hit me up! I would love to include it in my next edition😎
Ankit Rathi is a Cloud Data Technologist, published author & well-known speaker. His interest lies primarily in building end-to-end data/AI applications/products following best practices of Data Engineering and Architecture.