Confluent & Databricks join forces for AI data solutions

Confluent has partnered with Databricks to provide trustworthy real-time data for Artificial Intelligence (AI) applications.

Confluent and Databricks have announced the expansion of their partnership to integrate Confluent's Data Streaming Platform with Databricks' Data Intelligence Platform. This collaboration aims to empower enterprises by ensuring that real-time data can be effectively used for AI-driven decision-making.

The integration will enable businesses to build AI applications more efficiently by connecting Confluent's Tableflow with Databricks' Unity Catalog. This seamless governance of data across operational and analytical systems is expected to address the challenges faced by enterprises in building AI applications due to data silos.

"For companies to maximise returns on their AI investments, they need their data, AI, analytics and governance all in one place," said Ali Ghodsi, Co-Founder and CEO of Databricks. "As we help more organisations build data intelligence, trusted enterprise data sits at the centre. We are excited that Confluent has embraced Unity Catalog and Delta Lake as its open governance and storage solutions of choice, and we look forward to working together to deliver long-term value for our customers."

Jay Kreps, Co-Founder and CEO of Confluent, highlighted the importance of real-time data for AI. "Real-time data is the fuel for AI," said Kreps. "But too often, enterprises are held back by disconnected systems that fail to deliver the data they need, in the format they need, at the moment they need it. Together with Databricks, we're ensuring businesses can harness the power of real-time data to build sophisticated AI-driven applications for their most critical use cases."

The partnership between Confluent and Databricks addresses the existing challenges enterprises face in merging operational data with analytical insights. Only 22 percent of enterprises reportedly feel confident that their current IT infrastructure supports new AI applications, with a significant gap between operational and analytical systems being identified as a hurdle.

One of the key components of the new partnership is the bidirectional integration of Confluent's Tableflow with Delta Lake and Databricks' Unity Catalog. This integration promises to deliver consistent, real-time data that is both secure and trustworthy, across different systems. The inclusion of custom integrations will ensure that metadata, which is critical for AI applications, is automatically applied.

Dr. Dora Simroth, Head of Data and AI Engineering at E.ON Digital Technology, commented on the integration's significance. "Leveraging proximity to generation sources is a key factor not just in the energy sector, but also in the field of data," said Dr. Simroth. "Confluent and Databricks are already essential technologies in our Data and AI stack. These integrations will allow our practitioners to work on a single source of well-defined and timely data for both our operational and analytical plane. By partnering, Confluent and Databricks open up a faster path for us to build data and model-centric digital solutions."

The enhancements in the partnership ensure that operational data from Confluent becomes more accessible within Databricks, while Databricks data can be utilised across various processors within enterprises. This improvement allows AI applications and data analytics to derive consistent insights from the same real-time data, bolstering AI-driven decision-making processes.

The integration is anticipated to facilitate seamless interaction between enterprise applications, analytics, and governance systems, which is seen as critical for scalable AI innovation. The collaboration between the two companies underscores an industry trend towards bridging the gap between operational and analytical data for AI optimisation.