What does it really mean today to say, “every business is a data business”?

“It goes without saying that in today’s digital first economy businesses are more reliant on data than ever before. But not all businesses can claim that they’re fully in control of their data resources, able to make queries that will inform decision making and drive innovation. Executing a successful data strategy is dependent on having immediate access to data assets, supported by self-service capabilities that will help businesses to drive their own data insights.

“In practice, this means businesses need to begin integrating data across their cloud and on-prem resources, using an infrastructure that will allow them to rapidly integrate any new sources as they build out their cloud architecture. It’s important to have this data foundation in place to create the right environment for artificial intelligence (AI) and machine learning (ML). AI and ML projects require rapid iteration and need the ability to quickly find and integrate new data sets. Business driven AI and ML projects will deliver greater tangible results with faster access to enterprise data assets, regardless of the source. ”

In your view, what separates enterprises that succeed with data-driven innovation from those that stagnate?

“The businesses that will stand apart are those that build data infrastructures that span silos, regions and business units, allowing them to query data no matter where it resides. This will improve access, ease of use, data reliability and trustworthiness, and create new opportunities for cross-functional sharing. To be truly data driven, businesses need to simplify processes and place data more directly into the hands of those who use it.

“This can be achieved by embracing the concept of data products, a set of tools that empower business functions to solve problems quickly and with agility. Data products span multiple data resources, enabling businesses to transform, curate and share business critical data sets within minutes. Data products make data more accessible to non-technical users, reducing complexity and the need for long conversations and multiple iterations, while speeding the time to insight. ”

How is the definition of “speed” changing in the age of real-time data and customer expectations?

“We’ve seen the difference it can make firsthand, working with large enterprises that were reliant on platforms that took hours to perform data analytics queries, which led to inconsistencies and delayed insights. They resolved this by adopting managed frameworks that enabled them to accelerate data processing and queries, reducing the time needed to generate data insights to just a few minutes. HSBC was able to execute queries up to 20X faster, drastically reducing data transfers and duplication, resulting in significant operational and efficiency gains and cost reductions. This type of improvement enables organizations to respond more quickly to business needs, adapt to market changes, and make data-driven decisions faster.

“Recent innovations have also made it possible to autonomously identify and cache the most used, or most relevant data based on usage patterns, to dramatically improve the performance of enterprise data teams. This process of smart indexing and caching can autonomously accelerate interactive workloads by an average of 40%. ”

Where do you see the most significant opportunities for rapid innovation through data in your industry?

“Up until now, organisations have been centralising data to support decision-making. They’ve achieved this by focusing on known data with a clear model, and well-understood questions, using what we term ‘known data’ to answer what we refer to as ‘known questions.’ While still a valid approach, this model fails to enable data exploration, experimentation and the evolution of the data-driven business strategy.  

“Opportunities lie in grasping what is referred to as ‘unknown data’, which exists either within or outside the organisation, and which hasn’t previously been known. Unknown data may reveal business value that hasn’t been discovered yet that could be unique to the business. Unknown questions are questions that have yet to be asked, which may support innovation and potentially lead to significant competitive differentiation in your respective industry.”

“These new models can be developed using data-driven innovation, a strategic approach that leverages the power of data to facilitate better decision-making. It recognises the need for high-quality, well-organised data and a carefully planned data management strategy and business strategy, and how ultimately data analytics support an organisation’s business model.”

What role does leadership culture play in creating organisations that can innovate quickly with data?

“Leadership is a critical piece that we see consistently across all of our customers. Data is an enterprise asset that spans across organisations and requires every leader to establish a data-driven culture that supports their business, as well as is governed and secured in a manner that is consistent with their business.”

Which emerging technologies are proving most important for enterprises competing on speed?

“AI and ML are the most obvious use cases. AI systems learn and make decisions based on the data they are trained on. Without high-quality, representative, well-organised big data, AI models cannot function effectively. However, the majority of enterprise organisations are hampered by fragmented data that is slow and expensive to access. AI systems need access to the largest, freshest and most available datasets.

“While speed is integral to managing AI data stacks, businesses need to be mindful of AI data governance to ensure that that data remains discoverable, accessible, and properly controlled wherever it resides. Effective AI governance frameworks clearly define data ownership and stewardship, and standardise metadata, lineage tracking, and access controls. Instead of slowing down innovation, governance accelerates it by making high-value, trustworthy data easily accessible for AI and analytics.”

How can enterprises avoid getting bogged down in data complexity while trying to move faster?

“Starburst recently co-sponsored a new market research report led by the Boston Consulting Group, which found that enterprise architectures are stretched to the limits, with more than 50% of data leaders highlighting architectural complexity as a significant pain point. The report also found that the total cost of ownership of data is set to double because of increased spending on compute resources and the costs associated with retaining and recruiting the right talent to manage data initiatives. The report notes that: “many companies find themselves at a tipping point, at risk of drowning in a deluge of data, overburdened with complexity and costs.”

“Despite this, there is an appetite among data managers to boost investments and build new architecture based on data products that will allow them to address data-related challenges. This will allow them to reduce complexities while ensuring greater accessibility, increased reliability, and the ability to bring the data closer to the business context. This approach will lead to faster insights and better use of data.”

Looking three years ahead, what will define the fastest, most data-driven enterprises?

“As we look ahead over the next three years, enterprises are moving at a breakneck pace to not only future-proof their organisations to be data-driven but also AI-driven. What will be required is a data foundation and architecture that provides secure, governed access to all of the data. We are partnering with companies large and small to help them future proof their data foundations no matter what stage of their data journey they are in.”

Justin Borgman, Founder and CEO, Starburst.

Justin Borgman is a subject matter expert on all things big data and analytics. He founded Starburst in 2017, seeking to give analysts the freedom to analyse diverse data sets wherever they are located, without compromising on performance. Prior to founding Starburst, he was Vice President & GM at Teradata, where he was responsible for the company’s portfolio of Hadoop products. Justin joined Teradata in 2014 via the acquisition of his company Hadapt where he was co-founder and CEO. Hadapt created “SQL on Hadoop,” turning Hadoop from a file system to an analytic database accessible by any BI tool.