DDrona4U
Sign inCreate account
My learningData Bricks Interview QuestionsLesson 20
Small Files in Delta Lake: Every Lever You Can Use to Fix Read Performance

Lesson 20

Small Files in Delta Lake: Every Lever You Can Use to Fix Read Performance

In Databricks Delta Lake, small files hurt read performance by increasing metadata overhead, file-open costs, and task scheduling inefficiency. To fix this, engineers use techniques like OPTIMIZE compaction, auto-compaction, optimized writes, proper partitioning, Liquid Clustering, batching streaming writes, tuning file sizes, and reducing excessive parallelism to create larger, more efficient Parquet files for faster query execution.

Get the full lesson

Sign in to unlock everything beyond the preview — it's free.

  • Take timestamped notes as you watch
  • Read the full transcript and download resources
  • Join the discussion and track your progress
Sign inCreate free account

Curriculum

20 lessons · 45m

0/20 lessons done45m left
  1. 0102:00

    Lakehouse vs Data Lake + Data Warehouse: What’s the Real Difference?

    02:00

  2. 0201:59

    What Happens Between the Control Plane and Data Plane When You Run a Databricks Notebook?

    01:59

  3. 0302:01

    Why Databricks Runs in Your Cloud Account — And Why It Matters for Security

    02:01

  4. 0402:02

    How Databricks Separates Storage from Compute — The Core Architecture Behind Its Scalability

    02:02

  5. 0502:29

    What Is a Databricks Workspace and How Does It Connect Clusters, Jobs, Catalogs, and Users?

    02:29

  6. 0602:52

    Databricks Workspace: The Central Hub for Users, Clusters, Jobs & Catalogs

    02:52

  7. 0702:13

    Databricks Runtime vs Open-Source Spark: What Do You Lose When You Leave?

    02:13

  8. 0802:21

    Databricks Serverless Compute: What Happens When There’s No Cluster to Manage?

    02:21

  9. 0902:11

    Delta Transaction Log: How ACID Works on Top of Parquet Files

    02:11

  10. 1002:01

    Delta Lake Time Travel: Recovering a Table After an Accidental Truncate

    02:01

  11. 1102:12

    OPTIMIZE vs VACUUM vs ZORDER in Delta Lake: When Should You Use Each in Production?

    02:12

  12. 1202:20

    Liquid Clustering vs Z-Order vs Partitioning: Best Strategy for a 50 TB Multi-Filter Table

    02:20

  13. 1302:15

    Deletion Vectors vs Copy-on-Write: How Delta Lake Speeds Up MERGE and DELETE Operations

    02:15

  14. 1402:32

    Delta UniForm: Querying Delta Tables from Iceberg and Hudi Without Data Rewrites

    02:32

  15. 1502:22

    Delta Lake Schema Evolution: What’s Automatic, What Needs mergeSchema, and What Can Break Your Pipeline?

    02:22

  16. 1602:11

    Concurrent Writes in Delta Lake: What Happens When Two Streaming Jobs Hit the Same Table?

    02:11

  17. 1702:08

    Managed vs External Tables in Databricks: Which One Should You Choose in 2026?

    02:08

  18. 1802:09

    VACUUM with 0-Hour Retention: Why Delta Lake Time Travel Suddenly Broke

    02:09

  19. 1902:13

    Zero-Downtime Migration: Converting a 10 TB Parquet Table to Delta Lake Safely

    02:13

  20. 02:16

    Small Files in Delta Lake: Every Lever You Can Use to Fix Read Performance

    02:16