What is a data ecosystem? with insights from a data scientist!
I am having a coffee at Hong Kong airport, on my way back to Melbourne after an eventful week in the Netherlands filled with productive meetings, presentations, and guest lectures. The trip has provided me with valuable insights that I'd like to reflect upon. During my visit to the University of Utrecht, I had the opportunity to deliver a guest lecture on the governance challenges faced by international sporting bodies in the digital age. At the World Congress on Science and Football in Groningen, I actively participated in a panel discussion focused on the impact of the digital revolution on sports marketing. Additionally, at the National Sport Centre Papendal, I had the chance to catch up with my colleagues from digital marketing agency Techonomy, and I also spoke to a group of students from the Amsterdam University of Applied Sciences about what makes a great ‘digital transformation’ leader in sport. Throughout these conversations, the concept of 'digital ecosystems' emerged as a recurring topic. Before delving into the application of this concept to the digital world, let's first explore the original notion of an ecosystem.
Ecosystems are captivating and intricate systems that encompass the dynamic interplay between living organisms and their physical environment. They serve as the stage upon which the intricate web of life unfolds, showcasing the delicate balance, dependencies, and processes that sustain diverse life forms. They consist of two primary components, the biotic and abiotic components. The biotic components encompass the living organisms within an ecosystem, including plants, animals, micro-organisms, and fungi. They interact with one another, forming complex relationships such as predation, competition, symbiosis, and mutualism. The abiotic components comprise the non-living elements of an ecosystem, including physical factors such as sunlight, temperature, rainfall, soil composition, topography, and air quality. These abiotic factors shape the structure and function of the ecosystem, influencing the distribution and behaviour of the biotic components.
Ecosystems provide numerous benefits that are essential for human well-being such as goods and resources such as food, freshwater, timber, medicinal plants, and fuel, sustaining human livelihoods and economies. They regulate climate, water quality, air purification, soil fertility, disease control, and natural disaster mitigation, thereby supporting human health and resilience. Ecosystems also provide aesthetic, spiritual, and recreational value, fostering cultural identity, inspiration, and opportunities for relaxation and education. And finally, ecosystems underpin the functioning of other services, such as nutrient cycling, pollination, and soil formation, which are critical for the long-term sustainability of ecosystems and human societies.
Drawing inspiration from natural ecosystems, we can apply the concept of ecosystems to the digital realm, specifically to the lifeblood of organisations—data. Data ecosystems represent the dynamic interconnections between data, technology, and human resources. They serve as the foundation for data-driven decision-making, enabling organisations to navigate the complexities of the digital age. Data ecosystems consist of various components that collectively enable the flow, storage, processing, analysis, and application of data: data-sources, -infrastructure, -professionals, -translations, -applications, and -governance/security. At the National Sport Centre I also caught up with data scientist Daan Luttik, who is the Chief Technology Officer at Techonomy. As a data practitioner, Daan emphasised the importance of acting on data insights to drive value, rather than just relying on intuition. He noted that there are two questions that you need to ask yourself when it comes to data-processes:
1. "How can I use data to automate or improve my processes directly", for instance use a combination of business-rules and classification models to automatically mail customers about a part of your business-proposition that is relevant for them but that they are not utilising.
2. "Which insights will actually drive me to take different actions or decisions", for instance a vanity metric like "all-time total sales" will probably not change your actions, while a notification of a 10% increase in churn will probably be a strong case to divert internal resources.
It is often useful to view all applications in the ecosystem not just as a data-source, but also evaluate how those applications, and the processes that they facilitate can benefit from more, different or enriched data. This is often what drives the real change.
Data sources include the diverse origins of data, such as sensors, mobile devices, databases, applications, social media, and external data providers. These sources generate vast volumes of raw data, forming the basis of the data ecosystem. The data infrastructure includes the technical systems, software, platforms and frameworks that support the storage, integration, and processing of data. It comprises databases, data warehouses, data lakes, cloud platforms, business-rules, AI-models, and networking infrastructure. Of extreme importance are the data professionals, such as data engineers, data scientists, and data analysts, who play integral roles within the data ecosystem. They bring specialised skills and expertise to handle, analyse, and derive insights from data. One of the main survival mechanisms of a healthy data ecosystem would be the underpinning governance structures that ensure the responsible management, privacy, and ethical use of data within the ecosystem. Good data governance will be underpinned by data policies, standards, data quality, compliance, and security measures. Luttik again:
Data-quality is of paramount importance. It makes the difference between fully adopted dashboards, flawless, automated applications, and the ability to make bold decisions from those insights when the quality is there, as opposed to barely used or untrusted dashboards and applications which in turn result in extra manual work, shadow-administrations and misalignment due to arguments over the fundamental facts when the quality is lacking.
Similar to natural ecosystems, data ecosystems also exhibit hierarchical levels of organisation and interdependence. First data is collected from various sources and integrated into a centralised repository or data lake. This raw data forms the foundation for further processing and analysis. Data then undergoes centralisation followed by transformation, cleaning, and aggregation to make it ready for analysis. Data engineers develop the processing and storage systems in databases or data warehouses, making it accessible for querying and analysis. Data scientists and data analysts then employ statistical techniques, machine learning algorithms, and visualisation tools to derive meaningful insights from the data. They uncover patterns, trends, and correlations, enabling informed decision-making. Insights gained from data analysis are then utilised by functional experts (such as sport marketeers), managers and other decision-makers to drive business strategies, optimise processes, and enhance customer experiences.
However, as Daan Luttik argues:
One notable distinction between natural ecosystems and data ecosystems is that in regard to data you can really think about hierarchies in terms of data-lineage or access (data lineage is the process of understanding, recording, and visualising data as it flows from data sources to consumption). Having a clear vision when it comes to this provides consistency, clarity of purpose and clarity of roles. For instance, At Techonomy we tend to work with a bronze-silver-gold model, where the raw data is cleaned and stored in a bronze model. Data-sources are connected into central concepts like "customer" or "sale" in the silver layer (which is usually presented to the customers internal data-analysts) and the gold layer contains the application specific data which is usually aggregated. The last "gold" layer in turn can be separated into different "data products" to further clarify the purpose and control.
Well-designed data ecosystems provide numerous benefits that are crucial for organisations and society at large. They empower organisations to make data-driven decisions, leveraging insights to enhance operational efficiency, identify opportunities, and mitigate risks. At the highest level of efficiency are the autonomous data-driven systems, where business-rules, machine-learning, business-processes and application design come together to create something that is better (and much faster) than a human could ever do themselves. This opens the door for professionals to have time and mental space for an even higher level of creativity and strategic thinking. Luttik says in that regard that:
It used to be that only developers could automate their own job away. Now with the availability of data and products with automations and no-code functionality other functions like marketing can do the same. This enables them to think in terms of strategy and processes rather than single touch-points and emails and focusing on the cutting-edge rather than the status quo.
By harnessing the power of data, organisations can uncover new market trends, customer preferences, and innovative solutions, gaining a competitive edge. Well-designed data ecosystems can also enable organisations to understand customer behaviours, preferences, and needs, facilitating personalised experiences and targeted marketing strategies. By analysing big data sets, and with the help of data scientists, organisations can identify inefficiencies, streamline processes, and optimise resource allocation, leading to cost savings and improved productivity. At a more sophisticated level well designed and equipped data ecosystems enable organisations to develop predictive models, foresee trends, anticipate customer demands, and make proactive business decisions. Daan Luttik provides some interesting examples.
When it all comes together you can achieve amazing results. For instance, marketeers at multifunctional sports and entertainment venue Rotterdam Ahoy in the Netherlands have set system rules to intervene automatically in various communication scenarios. This is based on their data warehouse feeding the most important systems with up-to-date information. For example, the marketing technology can design and send personalised communications to upsell parking tickets to those who haven’t bought them yet. Another example is Campercontact, a freemium camper services platform. Based on insights generated about the customer journey they realised that many more paid memberships could be sold by simply sending an automated trigger to upgrade when a user reached any of the limits of the free content tier.
In the end, the success of both a natural and data ecosystem hinges on effective collaboration and communication. In a natural ecosystem, scientists, researchers, and conservationists collaborate to share findings, coordinate efforts, and develop conservation strategies. Similarly, in the data ecosystem, data engineers, data scientists, and data analysts work collaboratively, sharing knowledge, exchanging insights, and fostering synergy to derive maximum value from the data.
Reflecting on the various talks and discussions I participated in during my week in the Netherlands, I realised the usefulness of the ecosystem metaphor in representing the interconnected network of data, technology, and human expertise. The data ecosystem mirrors the intricate balance found within natural ecosystems. By understanding the components, functions, and importance of data ecosystems, leaders, managers, and governing bodies of organisations, including those in the sports industry, can harness the power of data more effectively. This understanding enables organisations to deeply engage with customers, drive product and service innovation, excel in sophisticated marketing campaigns, and make well-informed decisions in an increasingly data-driven world.