Data Leaders’ Best Practices to Overcoming Key Data Integration Challenges

The enterprise data integration skills shortage

The greatest challenge for enterprise data integration capabilities today is a lack of staff with the right skills. Of the executives we surveyed, 82% say securing the right staffing resources is at least ‘quite challenging’, with 44% describing it as ‘very challenging’.

“Developers are very costly,” notes Power. “They’re scarce and they’re expensive.”

At the same time, 77% say processing high volumes of data is at least ‘quite challenging’, while 73% say their teams find dealing with multiple sources of data and complex data formats at least ‘quite challenging’.

Similarly, 81% say ensuring the quality of this data as at least ‘quite challenging’ and 73% say a lack of technology or services to facilitate data integration is creating challenges for their teams.

Difficulty building real-time data pipelines is another hurdle faced by many, with 78% of respondents describing this as at least ‘quite challenging’.

ONS Chief Data Architect Tomas Sanchez concludes: “It’s no secret that the UK government has been wanting to improve data sharing and data interoperability across departments for a very long time now. Now, there is real momentum towards achieving that ambition.”

Overcoming barriers to effective data integration

Our research has identified two main strategies for overcoming the barriers to effective data integration enterprises are grappling with today.

On the organizational side, data-focused executives are working to ‘join up’ the data their various business units and teams are looking after. To do this, they’re developing shared data platforms and organizational structures geared toward breaking down data silos.

“We’ve been discussing with our partners across government, and we’ve put forward a specific proposal for how to create a platform to achieve that data sharing,” Sanchez says. “Rather than have departments talking to each other to exchange data for their project needs, we will have a platform to facilitate that sharing of data for specific research projects.”

“WPP has launched a new organization called Choreograph,” adds Vipul Parmar, Global Head of Data Management at WPP. “What it does is draw upon our ‘data giants’, as well as some of our smaller agencies and the pockets of resources that have been effectively working in silos, to support clients by doing the same things.

“This was an important step toward breaking down those silos, bringing those people together under one umbrella organization.”

On the technical side, executives are also investing in technologies that allow non-data scientists to handle some data integration tasks.

“Low-code, no-code means that a savvy business user from one of our business lines can become a configurator,” Power concludes. “It’s not a Nirvana, but it does tend to shorten the cycles. We are looking to embrace low-code/no-code everywhere.”

Further investment in technologies like these may be necessary to address the data integration challenges facing businesses today. Just 58% of our survey respondents agree they have the right tools to access data, process it and put it where it needs to be.