This episode of The Road to Intelligent Data Apps, focuses on Databricks intelligent data apps with host Shelly Kramer, managing director and principal analyst at theCUBE Research, who is is joined by analyst George Gilbert and Jason Pohl, Senior Director of Data Management for Databricks for a conversation about Databricks on intelligent apps, and a look at what’s now and what’s next for the company.
Watch or listen to Databricks on Intelligent Data Apps with Jason Pohl here or stream it wherever you get your podcasts:
This is part of our continuing conversation focused on what we see as the Next Data Platform, an emerging data platform where the leading vendors are Databricks, Snowflake, AWS, Azure, and Google. If you’re new to this conversation, what we’re seeing in this evolution is that the data platform is actually becoming an application platform, as all applications are becoming driven by data, analytics, and AI.
Databricks, Intelligent Data Apps and the Evolution of the Modern Data Stack
For our discussion today, we’ll take a closer look at Databricks, both the company roadmap and customer activity, and the emerging Databricks tech stack. We’re thrilled to have Jason join us for a conversation about Databricks and intelligent data apps.
First, some historical context: Snowflake defined the modern data stack with the separation of compute from storage, which essentially allowed the management of infinite amounts of data in a data lakehouse architecture.
The data lakehouse gave customers a simple solution for managing their end-to-end data analytic data estate. And here’s where it gets interesting: We see Databricks redefining Snowflake’s core innovation of separating storage from compute — and that’s part of the reason we were excited about this conversation with Jason. Our discussion covered a variety of things, including:
- An overview of the Databricks’ version of the data lakehouse, in which storage and compute can come from different vendors.
- The process by which end-to-end data management shifts to end-to-end data governance in the Databricks Unity catalog.
- A key differentiator for Databricks is that different compute engines, including Snowflake, can run different workloads on the data, providing customers with a much wider price, choice and performance or capabilities.
- An exploration of what a data lakehouse is and how it differs from a more traditional data platform (e.g., multi-vendor compute versus single vendor, governance separate from DMBS, etc.), as well as what the ability to write data in open format for multiple engines means for customers.
- An overview of how Databricks’ Intelligent Lakehouse solution works and how that benefits customers.
- This is a discussion of workloads and what distinguishes them, an exploration of where customers are in deployment and production of their AI workloads, and thoughts on how customers can leverage the tools and pipelines in traditional ML workloads in building their Gen AI functionality.
Other topics covered in this discussion with Databricks’ Jason Pohl include:
- Application platforms and how we expect the definition of apps will change over time. How this impacts how devs build apps and operationalize analytics, and the role of workflow and transaction support.
- There was much discussion around data governance and its key role, as well as multi-vendor data estates and how and when customers are using compute engines from multiple vendors.
- Databricks’ Unity Catalog and the growth capabilities they envision there over time, as well as how customers plan to implement Unity.
We wrapped the episode with a conversation about Databricks intelligent data apps with a look ahead at the company’s vision for the Intelligent Lakehouse and what we collectively see ahead for the industry. Jason shared a wealth of valuable information, and we expect to revisit this conversation later in the year as we continue to track Databricks’ on intelligent data apps and the progress the company is making on this front.
