Data-Driven Decision-Making: Trends, Challenges, and Solutions

As organizations scramble to tap into the immense value of all the data available to them, they also struggle with a growing set of challenges.

In today’s digital economy, data-driven decisions are rapidly becoming the norm. According to a 2023 survey by Drexel University’s LeBow College of Business, 77% of data and analytics professionals say that data-driven decision-making is a leading goal for their data programs. Yet fewer than half rate their ability to trust the data used for decision-making as “high” or “very high.”

That trust gap has profound implications.  In a separate survey conducted in 2022, for example, S&P Global Market Intelligence found that fewer than one-quarter of respondents said nearly all strategic decisions are data-driven within their companies. That reveals an important opportunity for improvement.

The vast majority of organizations recognize that they need to facilitate a data-driven approach, especially for strategic decisions. When asked whether data will be more important to their organizations 12 months from now, the same group showed strong indications that the data-driven approach is on the rise. Only about four percent of respondents said that data will actually become less important in their decision-making processes over the next year.

Data Governance Gains Momentum

To get the most from data analytics initiatives, organizations must proactively work to build data integrity. Doing so requires a sound data governance framework.

Data governance aims to gain control of the various elements that influence trust and confidence in data.  It also provides a structure for managing an ever-increasing body of risks and requirements around security, privacy, and regulatory compliance. Although most organizations recognize that effective data governance is critical, the majority are nevertheless still in the early stages of maturity with their data governance initiatives.

Some are further along in the process than others. Several industries have already widely adopted mature governance programs. Transportation and financial services are leading the way, for example, due in large part to their relatively intensive compliance requirements.

The aforementioned LeBow study set out to learn more about how today’s data professionals view data governance, their biggest challenges and barriers to success, and the differing views of governance within their respective organizations. The authors surveyed more than 450 data and analytics professionals across a range of industries, serving in a variety of different roles. Respondents’ functional titles included everything from C-level executives to line-of-business managers, IT executives, data stewards, data architects, data managers, and data analysts.

Respondents listed operational efficiency and compliance as top priorities for data programs, after the #1 priority of strengthening their ability to make effective data-driven decisions. Nevertheless, most are still in the early stages of maturity. Moreover, they differ widely in organization, operations, and funding.

The report concluded that there are reliable, data-driven reasons why companies should invest in building or maturing their data governance programs. The topmost value-generating benefit, according to respondents with mature programs, is the ability of such initiatives to strengthen overall data quality. Data quality and governance are key components in building data integrity; they work together to enable business users to understand and trust their data for confident decision-making. As such, data governance is a key factor in determining how well organizations achieve compliance and trust.

Enterprise Challenges in 2024 and Beyond

While data governance provides a foundational framework through which to approach data integrity, the big-picture process of building data integrity calls for a systematic approach combining technology tools, internal change management, and a company-wide commitment to results.

The motivations for achieving high levels of data integrity are shifting. In the post-pandemic era, agility and long-term resiliency are paramount. Improving business processes, rolling out automation, increasing agility of decision-making, and enhancing customer service and engagement top the list of priorities.

Technical Challenges

Many of the top barriers to achieving those objectives, notes S&P Global Market Intelligence’s Senior Analyst Paige Bartley, tend to be technical in nature. Data quality and consistency, for example, are essential prerequisites for trusted data-driven decisions. Privacy and security are increasingly under the spotlight, driving an increased focus on regulatory compliance and data governance.

Just over three-quarters of all respondents in the LeBow survey stated that improving data quality and trust is the single most important goal for advancing their data programs. The most common business case driving data quality and governance initiatives is a desire for better business decisions. Executives are increasingly insisting that the people in their organizations have access to data-driven insights to support their decisions with reliable, trustworthy data.

Seventy-seven percent of respondents in the LeBow survey indicated that improved decision-making capabilities are the leading driver of their data programs, followed by improved operational efficiency (73%), cost reduction (62%), revenue generation (59%), regulatory compliance (57%), and risk mitigation (57%).

Most enterprises also recognize the value of unifying their data, providing a holistic view across various departments and divisions. That requires strong data integration capabilities, but it leads to a better capacity for building usable data sets for data science and AI/ML models. It also enables a 360° view of customers, bringing together multiple data sources under one roof. Ultimately, that translates to better customer service, stronger product management, and more efficient marketing spend.

Privacy and Security Challenges

Privacy and security also emerge as key concerns. Most organizations struggle to maximize the value of their data while also remaining hyper-aware that regulatory pressure and customer expectations demand a consistent and systematic approach to the privacy and security of personal information. Structured data governance programs, along with a solid technology foundation to enable company-wide adoption, provide a solution to those challenges.

Data Quality Challenges

Data quality remains a top concern for many enterprises. Even so, the nature of those concerns is shifting. As the use of data expands to cover a broader range of users and use cases, data quality issues are more apparent than ever. Legacy data quality issues remain a problem at many enterprises, even as new challenges begin to show up on the radar.

As the volume, velocity, and variety of data increases, it has become harder than ever to keep up with the challenge of poor data quality. The old way of thinking dictated that an organization clean up its data periodically. In practice, that was never an adequate approach to the problem. To achieve and maintain high levels of data quality at scale, a proactive and systematic approach is necessary. That means having the right technology to catalog data assets, define common terminology, establish clear business rules and data owners, and proactively alert the right people when potential problems emerge.

The mounting problem of poor data quality isn’t just a technical challenge; it also requires a commitment to cultural change. Data democratization means that there are more users consuming, including business users with relatively low levels of technical expertise. Enterprises must implement systems to support those employees but must also affect a supporting cultural shift.

Data Governance Drives Confident Decisions

Historically, many observers have regarded data governance as a supporting framework for regulatory compliance. Most enterprises are increasingly aware, however, that data governance plays a broader role in supporting confident data-driven decisions. Initially an afterthought in many organizations, data governance now occupies a strategic role. Without it, organizations simply cannot adopt a data-driven approach to decisions at scale.

When S&P Global Market Intelligence asked organizations about the benefits they’re seeing from their data governance initiatives, the top two responses had nothing to do with compliance and everything to do with trusted decisions. Top executives see governance as a key enabler for faster access to relevant data and a framework for facilitating higher-quality data and insight across the entire organization. By thoughtfully implementing a data governance program and supporting it with both technical and human resources, enterprises are improving data-driven decisions for everyone within their respective organizations.

Emily Washington, Senior Vice President of Product Management at Precisely, notes that our clients are consistently successful when they anchor their data quality and data governance initiatives to well-defined business objectives. By providing clarity and transparency around enterprise data, an organization builds confidence and instills a common purpose among users throughout the company.

A consumer products company might begin by establishing clear objectives around customer satisfaction, upsell/cross-sell, and retention rates. To achieve that, the data governance team might work with product marketing, product management, customer service, and sales to understand their specific needs with respect to having a 360° view of each customer. That could lead to defined KPIs around data quality, data integration, and data enrichment.

By focusing on clearly defined business outcomes, owners and sponsors of the data governance initiative can achieve quick wins, celebrate success, and enlist users throughout the organization to embrace the cultural shift toward trusted, data-driven decisions. Many teams choose high-value use cases that offer maximum impact in a minimal timeframe. From there, it becomes easier to move on to new use cases, building credibility and increasing forward momentum.

Ultimately, data governance offers a structured approach to unifying the management and use of data across an entire organization. Naturally, that still incorporates privacy and security, but more importantly, it encompasses the people, processes, and technology necessary for organizations to optimize their use of data and successfully adopt a data-driven approach to decision-making.

Best Practices from Mature Data Governance Organizations

Experts at the LeBow Center for Business Analytics divide organizations into five levels of program maturity, ranging from occasional and inconsistent performance of data governance as part of individual projects to comprehensive data governance programs optimized and aligned with a process of continuous improvement.

Companies with mature data programs are implementing a broad range of strategies in parallel.  Data integration topped that list, as organizations strive to eliminate data silos and achieve a holistic view of the information that matters most. Leading companies are also implementing technology and processes to profile data and improve data quality in a scalable way.

Many companies are also recognizing the value of investing in their people.  Forty percent of respondents in the LeBow study say they are working to improve data literacy skills, bolster competencies among data specialists, and increase staffing and resources for data management.

Data governance is also a high priority among leading companies.  Experts at LeBow have concluded that organizations with the most mature and effective data governance programs tend to have a higher incidence of dedicated budgets to support those programs.  41% of respondents in the 2023 study say that data governance is among their top priorities.

The authors of the LeBow study also identified two emerging priorities.  Data enrichment adds valuable context, increasing the overall effectiveness of data programs by enhancing data-driven decisions. Spatial analytics, likewise, appeared on the list of top priorities. As companies begin to understand the immense power of geospatial context, LeBow predicts that more and more organizations will invest in location intelligence tools.

Precisely provides the world’s leading solutions for data integrity, including integration, data quality and governance, location intelligence, and data enrichment. As a sponsor of LeBow’s 2023 Data Integrity Trends and Insights report, Precisely is pleased to make the study available to our readers free of charge. Download it free today.