Hands-on Databricks Vector Search

Databricks Vector Search is now available in public preview, offering a serverless vector database for efficient similarity search. Powered by Databricks’ serverless compute infrastructure, Vector Search integrates with Delta tables, Unity Catalog, and Model Serving for seamless management and access. This article embarks on a simple, step-by-step exploration of this new feature through the process […]

Databricks Lakehouse Federation

Lakehouse Federation was announced at this year’s Data + AI Summit. The objective is to address data fragmentation and promote data governance across multiple systems. Lakehouse Federation lets us easily configure read-only connections to popular database solutions using drivers that are included on SQL Pro and Serverless warehouses, or Runtime 13.1 & up engineering clusters. […]

Liquid Clustering with Databricks Delta Lake

Databricks unveiled Liquid Clustering at this year’s Data + AI Summit, a new approach aimed at improving both read and write performance through a dynamic data layout. Recap: Partitioning and Z-Ordering Both partitioning and z-ordering rely on data layout to perform data processing optimizations. They are complementary since they operate on different levels, and apply […]

Third generation data platforms: The Lakehouse

Data Platform Evolution Initially, data warehouses served as first-generation platforms primarily focused on processing structured data. However, as the demand for analyzing large volumes of semi-structured and unstructured data grew, second-generation platforms shifted their attention towards leveraging data lakes. This resulted in two-tiers architectures with problematic side-effects: Complexity of maintaining and synchronizing the two tiers, […]