Conquering modern data stack complexities

How are data teams conquering the complexity of the modern data stack? Unravel Data has asked 350+ data scientists, engineers, analysts, and others who rely upon real-time data insights for decision-making to share their practices.

“For the third year in a row we’ve had the opportunity to take the pulse of enterprise data teams to better understand the daily challenges they face as they accelerate their ambitious big data analytics programs,” said Kunal Agarwal, CEO of Unravel Data.

“In just the course of a year we’ve seen a significant shift in how these growing, cross-functional teams are prioritizing DataOps as an established discipline across their organizations in a similar way that DevOps became an entrenched practice among software teams a decade ago. But despite this progress, this year’s survey also demonstrates that issues like FinOps, cloud utilization, and data security continue to present unique challenges to data teams,” Agarwal continued.

Data teams challenges

Cloud spending is now a critical KPI for the majority of data teams

More than two-thirds of data teams surveyed said that cloud spending has become a key performance indicator (KPI) of high strategic importance. When responses were broken down by role, almost 80% of business stakeholders said cloud spending was a critical KPI while 55% of data practitioners indicated the same.

Cloud resources are being underutilized

In addition to cloud spending being elevated as a top KPI, 44% of all respondents in this year’s survey also reported that they believe that they are leaving money on the table when it comes to their public cloud utilization. Alarmingly, 23% said they were unable to even estimate what percentage of their cloud resources went unused.

FinOps interest is high yet adoption lags

Despite the fact that data teams have reported a lack of visibility into cloud spending, the adoption of mature FinOps practice was not viewed as an immediate priority among respondents with just over 20% reporting that their data teams have an established FinOps practice while a third of data teams reported that they are still in the early planning phase of implementing FinOps.

DataOps as a practice is maturing

44% of respondents reported they are actively employing DataOps methodologies, compared to 21% of respondents in 2022, representing a 110% increase from the year prior. Further demonstrating the maturing DataOps practice, only 20% of respondents in this year’s survey said they were at the beginning stage compared to 41% last year.

Data reliability emerges as the top challenge

This year when participants were asked what they viewed as the top challenge with operating their data stack, 41% respondents cited the lack of data quality as their most significant obstacle, while 35% noted that the lack of visibility across their environments was the second biggest obstacle to managing their data stack.