Tag Archive

hadoop

What is RDD in Spark ? and Why do we need it ?

A developer-friendly guide to understanding Resilient Distributed Datasets (RDDs) in Apache Spark, their properties, and why they are fundamental for fast, fault-tolerant distributed computing.

2 min read

What is Apache HCatalog ?

A developer-friendly introduction to Apache HCatalog, its architecture, features, and how it fits into the Hadoop ecosystem.

2 min read

How-to : Write a CoProcessor in HBase

Coprocessor is a mechanism which helps to move computations closer to the data in HBase. It is like a Mapreduce framework to distribute tasks across the cluster.

2 min read

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read

Hive Strict Mode

![Sort By vs Order By vs Group By vs Cluster By in Hive]/assets/uploads/2015/01/images.jpg)

What is Hive Strict Mode?

Hive Strict Mode (hive.mapred.mode=strict) enables Hive to restrict certain performance intensive operations. Such as:

~1 min read

How-To : Configure MySQL Metastore for Hive ?

Hive by default comes with Derby as its metastore storage, which is suited only for testing purposes and in most of the production scenarios it is recommended to use MySQL as a metastore. This is a step by step guide on How to Configure MySQL Metastore for Hive in place of Derby Metastore (Default).

2 min read

Hadoop : Getting Started with Pig

What is Apache Pig?

Apache Pig is a high level scripting language that is used with Apache Hadoop. It enables data analysts to write complex data transformations without knowing Java. Its simple SQL-like scripting language is called Pig Latin, and appeals to developers already familiar with scripting languages and SQL. Pig Scripts are converted into MapReduce Jobs which run on data stored in HDFS (refer to the diagram below).

3 min read

Top 10 Hadoop Shell Commands to manage HDFS

So you already know what Hadoop is? Why it is used for? What problems you can solve with it? And you want to know how you can deal with files on HDFS? Don’t worry, you are in the right place.

2 min read
Back to Top ↑

bigdata

What does Skipped Stage means in Spark WebUI ?

Skipped Stages in Spark UI

You must have come across various scenarios where you see a DAG like below, where you see a few stages shows greyed out with a text (skipped) after the stage name. What does this mean? Did Spark ignore one of your stage due to an error? or this is due to something else? Well, it’s actually a good thing. It means that particular stage in the lineage DAG doesn’t need to be re-evaluated as its already evaluated and cached. This will save computation time for that stage. If you want to see what data frame for that stage was stored in the cache, you can check in Storage tab in Spark UI.

~1 min read

Dataframe Operations in Spark using Scala

A comprehensive, developer-friendly guide to common DataFrame operations in Apache Spark using Scala, with code examples and explanations for each join type.

2 min read

What is RDD in Spark ? and Why do we need it ?

A developer-friendly guide to understanding Resilient Distributed Datasets (RDDs) in Apache Spark, their properties, and why they are fundamental for fast, fault-tolerant distributed computing.

2 min read

What is Apache HCatalog ?

A developer-friendly introduction to Apache HCatalog, its architecture, features, and how it fits into the Hadoop ecosystem.

2 min read

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read

Hive Strict Mode

![Sort By vs Order By vs Group By vs Cluster By in Hive]/assets/uploads/2015/01/images.jpg)

What is Hive Strict Mode?

Hive Strict Mode (hive.mapred.mode=strict) enables Hive to restrict certain performance intensive operations. Such as:

~1 min read

How-To : Configure MySQL Metastore for Hive ?

Hive by default comes with Derby as its metastore storage, which is suited only for testing purposes and in most of the production scenarios it is recommended to use MySQL as a metastore. This is a step by step guide on How to Configure MySQL Metastore for Hive in place of Derby Metastore (Default).

2 min read
Back to Top ↑

spark

What does Skipped Stage means in Spark WebUI ?

Skipped Stages in Spark UI

You must have come across various scenarios where you see a DAG like below, where you see a few stages shows greyed out with a text (skipped) after the stage name. What does this mean? Did Spark ignore one of your stage due to an error? or this is due to something else? Well, it’s actually a good thing. It means that particular stage in the lineage DAG doesn’t need to be re-evaluated as its already evaluated and cached. This will save computation time for that stage. If you want to see what data frame for that stage was stored in the cache, you can check in Storage tab in Spark UI.

~1 min read

Dataframe Operations in Spark using Scala

A comprehensive, developer-friendly guide to common DataFrame operations in Apache Spark using Scala, with code examples and explanations for each join type.

2 min read

What is RDD in Spark ? and Why do we need it ?

A developer-friendly guide to understanding Resilient Distributed Datasets (RDDs) in Apache Spark, their properties, and why they are fundamental for fast, fault-tolerant distributed computing.

2 min read

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read
Back to Top ↑

big data

Unlocking Data Lakes with Apache Iceberg: The Open Table Format Revolutionizing Analytics

Why Apache Iceberg is the Game-Changer Your Data Lake Needs

Still relying on traditional data lakes with unreliable reads, lack of ACID guarantees, and painful schema evolution?

Welcome to Apache Iceberg—an open table format purpose-built for modern, petabyte-scale analytics. With native support for engines like Apache Spark, Trino, Presto, Hive, and Flink, Iceberg solves the biggest challenges in big data processing.

In this post, we’ll break down:

  • What Apache Iceberg is and why it matters
  • How its architecture works (with a visual!)
  • Key features that make it future-ready
  • Real-world use cases
  • A call to action for your next data platform decision
3 min read

How-To : Setup Realtime Alalytics over Logs with ELK Stack : Elasticsearch, Logstash, Kibana?

Once we know something, we find it hard to imagine what it was like not to know it.

— Chip & Dan Heath, Authors of Made to Stick, Switch

What is the ELK stack?

The ELK stack consists of open source tools ElasticSearch, Logstash, and Kibana. These three provide a fully working real-time data analytics tool for getting wonderful information sitting on your data.

3 min read

Hadoop : Getting Started with Pig

What is Apache Pig?

Apache Pig is a high level scripting language that is used with Apache Hadoop. It enables data analysts to write complex data transformations without knowing Java. Its simple SQL-like scripting language is called Pig Latin, and appeals to developers already familiar with scripting languages and SQL. Pig Scripts are converted into MapReduce Jobs which run on data stored in HDFS (refer to the diagram below).

3 min read

Top 10 Hadoop Shell Commands to manage HDFS

So you already know what Hadoop is? Why it is used for? What problems you can solve with it? And you want to know how you can deal with files on HDFS? Don’t worry, you are in the right place.

2 min read
Back to Top ↑

hive

What is Apache HCatalog ?

A developer-friendly introduction to Apache HCatalog, its architecture, features, and how it fits into the Hadoop ecosystem.

2 min read

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read

Hive Strict Mode

![Sort By vs Order By vs Group By vs Cluster By in Hive]/assets/uploads/2015/01/images.jpg)

What is Hive Strict Mode?

Hive Strict Mode (hive.mapred.mode=strict) enables Hive to restrict certain performance intensive operations. Such as:

~1 min read

How-To : Connect HiveServer2 service with JDBC Client ?

HiveServer2 (HS2) is a server interface that enables remote clients to execute queries against Hive and retrieve the results. The current implementation, based on Thrift RPC, is an improved version of HiveServer and supports multi-client concurrency and authentication. It is designed to provide better support for open API clients like JDBC and ODBC.

3 min read

How-To : Configure MySQL Metastore for Hive ?

Hive by default comes with Derby as its metastore storage, which is suited only for testing purposes and in most of the production scenarios it is recommended to use MySQL as a metastore. This is a step by step guide on How to Configure MySQL Metastore for Hive in place of Derby Metastore (Default).

2 min read
Back to Top ↑

java

Java : What does finalize do and How?

finalize method in the Object class is often a point of discussion regarding whether it should be used or not. Below are some important pointers on the finalize method:

2 min read

How-To : Generate Restful API Documentation with Swagger ?

Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services. The goal of Swagger is to enable client and documentation systems to update at the same pace as the server. The documentation of methods, parameters, and models are tightly integrated into the server code, allowing APIs to always stay in sync.

2 min read
Back to Top ↑

hdfs

How-to : Write a CoProcessor in HBase

Coprocessor is a mechanism which helps to move computations closer to the data in HBase. It is like a Mapreduce framework to distribute tasks across the cluster.

2 min read

Hadoop : Getting Started with Pig

What is Apache Pig?

Apache Pig is a high level scripting language that is used with Apache Hadoop. It enables data analysts to write complex data transformations without knowing Java. Its simple SQL-like scripting language is called Pig Latin, and appeals to developers already familiar with scripting languages and SQL. Pig Scripts are converted into MapReduce Jobs which run on data stored in HDFS (refer to the diagram below).

3 min read

Top 10 Hadoop Shell Commands to manage HDFS

So you already know what Hadoop is? Why it is used for? What problems you can solve with it? And you want to know how you can deal with files on HDFS? Don’t worry, you are in the right place.

2 min read
Back to Top ↑

Apache Iceberg

The Small-Files Problem: A Field Guide

The Maintenance Chapter Your Streaming Pipeline Was Missing

Your streaming job writes 2 MB Parquet files every two minutes. Six months later, your dashboard queries are ten times slower — and nobody changed the data. Sound familiar?

Yesterday, in Streaming from Kafka into an Iceberg Lakehouse, End to End, we built the whole pipeline — Kafka topic, Spark Structured Streaming, Iceberg table — and Step 6 ended with a warning: one-minute micro-batches create lots of small files — the classic streaming lakehouse problem. This post picks up exactly there. The pipeline works; the file layout is what’s rotting underneath it.

Same stack as yesterday: Kafka 4.3.1, Spark 4.2.0 (Scala 2.13), Iceberg 1.12.0, PySpark — and the same lakehouse.order_events table, partitioned by days(event_time).

In this post, we’ll:

  • See why small files hurt — with real numbers, not vibes

  • Diagnose the problem using Iceberg’s own metadata tables

  • Stop writing small files at write time

  • Fix the damage with compaction (rewrite_data_files, done right)

  • Keep the layout healthy with a checklist that fits on an index card

Step 1: Understand Why Small Files Hurt

It’s tempting to treat file count as cosmetic. It isn’t. Every file in your table is a tax paid four times: at plan time, at task-scheduling time, at request time, and at footer-read time.

Query planning. Iceberg query planning reads manifest files to decide which data files to scan — the manifest and snapshot layer I broke down in the Iceberg intro is exactly what gets stressed here. Dremio’s engineers measured this directly: the same dataset stored in 2,000 files of 256 MB each typically plans in under one second, while a table fragmented into 100,000 files pays a planning cost an order of magnitude higher. Manifests fragment too — every micro-batch commit adds manifests, and the metadata tree stops being an index and becomes a maze.

Task scheduling. Each file becomes at least one execution task. Scheduling costs roughly 1–10 milliseconds per task — trivial, until you multiply: a query against 100,000 files creates at least 100,000 tasks. Even if partition pruning eliminates 95% of them, the surviving 5,000 tasks carry 5–50 seconds of pure scheduling overhead before any data moves. On JVM engines like Spark, giant task graphs also pile on GC pressure, adding unpredictable latency.

Object-store requests. S3 charges $0.0004 per 1,000 GET requests. Reading a file takes at minimum one GET, but Parquet readers also fetch the footer and individual row groups — conservatively, two requests per file. So 100,000 files means 200,000 requests, or $0.08 per query, in storage API costs alone. A table queried 10,000 times a day burns $800 a day — $24,000 a month — on requests. Compact to 2,000 files and that drops to about $1,600 a month, a 93% reduction (source).

Footer reads. Every Parquet file carries a footer — schema, row-group metadata, column statistics. Reading it is mandatory and costs roughly the same whether the file is 1 MB or 512 MB: one to several milliseconds plus a network round trip. For a 256 MB file that’s noise; for a 1 MB file it dominates. If 95% of your read time goes to footers instead of column data, you have a severe small-files problem (source).

How fast it accumulates. Streaming engines checkpoint every 1–10 minutes, and each checkpoint commits new Parquet files — one per partition. A table with 100 partitions on 10-minute checkpoints produces 14,400 new files per day, most of them a few megabytes each (source). A table with 500,000 small files queried 50 times a day generates roughly 25 million GET requests a month. The write path that felt so elegant on day one is a liability by month six.

Step 2: Diagnose — Is This Actually Your Problem?

Don’t compact on vibes. Iceberg gives you metadata tables that answer the question directly:

-- File health at a glance: count, average size, smallest/largest
SELECT
  COUNT(*)                                        AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)     AS avg_mb,
  ROUND(MIN(file_size_in_bytes) / 1048576, 2)     AS min_mb,
  ROUND(MAX(file_size_in_bytes) / 1048576, 2)     AS max_mb,
  SUM(CASE WHEN file_size_in_bytes < 134217728
           THEN 1 ELSE 0 END)                     AS files_under_128mb
FROM lakehouse.order_events.files
WHERE content = 0;   -- 0 = DATA files; 1/2 = delete files

If a large share of files sits under 128 MB — or you see thousands of files averaging a few MB — you have the problem. Next, find where it hurts most, so your first compaction pass buys the biggest win:

-- Worst partitions first: file count and avg size per partition
SELECT
  partition,
  COUNT(*)                                     AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)  AS avg_mb
FROM lakehouse.order_events.files
WHERE content = 0
GROUP BY partition
ORDER BY file_count DESC
LIMIT 30;

Two more signals worth checking:

-- Are snapshots piling up from streaming appends?
SELECT snapshot_id, committed_at, operation, summary
FROM lakehouse.order_events.snapshots
ORDER BY committed_at DESC
LIMIT 20;

A wall of append snapshots with no replace (compaction) in sight confirms the diagnosis. And don’t forget the manifests metadata table — if manifest count is climbing with file count, planning overhead is compounding.

Outside SQL, the symptoms are just as telling: Spark UI jobs where a stage fans out into tens of thousands of tiny tasks, query plans that take longer than the scan itself, and — the one people notice last — your S3 GetObject/ListObjects request metrics climbing month over month while data volume stays flat.

Step 3: Fix It at Write Time

Compaction is corrective. The cheaper move is to stop writing small files in the first place. Three things worth knowing about the write path:

1. Target file size. The table property write.target-file-size-bytes defaults to 512 MB. Spark writers roll a new file when buffered data crosses this threshold — but only if a single task actually accumulates that much. A streaming micro-batch with 50 MB of data never reaches it, which is exactly how you get 2 MB files:

-- Set once at the table level (256 MB is a common streaming-table choice)
ALTER TABLE lakehouse.order_events
  SET TBLPROPERTIES ('write.target-file-size-bytes' = '268435456');

You can also override it per write with the target-file-size-bytes DataFrame option.

2. Distribution mode. write.distribution-mode decides how rows are shuffled before writing: none (no shuffle — each task writes whatever it has; the small-files factory), hash (hash-distribute by partition key), or range (range-distribute by partition or sort key). For streaming writes into a partitioned table, hash is usually the right call:

(df.writeStream
   .format("iceberg")
   .option("checkpointLocation", checkpoint_path)
   .option("distribution-mode", "hash")   # shuffle by partition key before writing
   .option("fanout-enabled", "true")      # keep partition writers open across batches
   .toTable("lakehouse.order_events"))

3. The fanout writer. Streaming data arrives out of order across partitions. Without fanout, every partition switch inside a task closes the current file and opens a new one — file rollover churn. write.spark.fanout.enabled (default false) keeps multiple partition writers open simultaneously, at the cost of more memory. For streaming jobs, enable it.

We used fanout-enabled: true in the streaming job from yesterday’s post for exactly this reason — without it, out-of-order events across days would have multiplied our file count from the very first batch.

Two more write-path habits that pay off:

  • repartition before batch writes. Line writers up with the partition spec so each task writes fewer, larger files:
from pyspark.sql.functions import to_date

def write_batch(batch_df, batch_id):
    (batch_df
       .repartition(to_date("event_time"))  # align writers with days(event_time)
       .writeTo("lakehouse.order_events")
       .option("fanout-enabled", "true")
       .append())

(query.writeStream
   .foreachBatch(write_batch)
   .option("checkpointLocation", checkpoint_path)
   .trigger(availableNow=True)
   .start())
  • Trigger intervals. The one-minute trigger we used yesterday is great for freshness demos and terrible for file layout — on a low-volume topic it guarantees tiny files. Prefer Trigger.AvailableNow or multi-minute processing-time triggers, and coalesce inside the batch function as above.

None of this eliminates compaction entirely — streaming tables will always drift — but it slows the drift from days to weeks.

Step 4: Compact What’s Already There

Yesterday’s post showed the one-line version — CALL lakehouse.system.rewrite_data_files(table => 'order_events'). Here’s the full picture. Iceberg’s compaction primitive rewrites selected data files into fewer, larger ones and commits the result as a new snapshot — fully online, no downtime; readers keep working against the old snapshot until the new one commits.

Binpack (default): fast and dumb, in a good way. It groups small files by partition and concatenates them up to the target size. No shuffle, no reordering — the cheapest possible rewrite:

-- Compact only the files that need it: small files in recent partitions
CALL lakehouse.system.rewrite_data_files(
  table    => 'order_events',
  strategy => 'binpack',
  where    => 'event_time >= current_timestamp() - INTERVAL 7 DAYS',
  options  => map(
    'target-file-size-bytes', '268435456',   -- 256 MB
    'min-input-files',        '5'             -- skip groups already healthy
  )
);

Sort: slower, but fixes layout too. Binpack doesn’t touch data ordering. If your queries filter on event_time and customer_id but files were written in arrival order, column statistics are weak and pruning suffers. The sort strategy globally sorts before allocating files to tasks, tightening min/max ranges per file:

-- Merge + cluster by the columns your queries actually filter on
CALL lakehouse.system.rewrite_data_files(
  table      => 'order_events',
  strategy   => 'sort',
  sort_order => 'event_time ASC NULLS LAST, customer_id ASC NULLS LAST',
  options    => map(
    'target-file-size-bytes',             '268435456',
    'partial-progress.enabled',           'true',
    'max-concurrent-file-group-rewrites', '5'
  )
);

-- For multi-column filters with no single dominant column, Z-order instead:
-- sort_order => 'zorder(event_time, customer_id)'

(Iceberg 1.12 also adds Hilbert-curve clustering as a newer alternative to Z-ordering — worth evaluating once your engine supports it.)

The options that matter in production:

Option Default What it does
target-file-size-bytes table’s write.target-file-size-bytes (512 MB) Output file size target
min-input-files 5 File groups with fewer files are skipped
min-file-size-bytes 75% of target Files below this are always candidates
max-file-size-bytes 180% of target Files above this are always candidates
max-concurrent-file-group-rewrites 5 Parallelism of the rewrite; raise for large tables
max-file-group-size-bytes 100 GB Splits the job into file groups
partial-progress.enabled false Commit file groups as they finish instead of one big commit
partial-progress.max-commits 10 Cap on commits when partial progress is on
rewrite-all false Force-rewrite everything, ignoring the filters above
rewrite-job-order none Order groups: bytes-asc, bytes-desc, files-asc, files-desc

Two of these deserve emphasis. partial-progress.enabled=true is the difference between losing hours of rewrite work on a mid-job failure and keeping what finished — always turn it on for large tables. And max-concurrent-file-group-rewrites is the knob real teams reach for at scale: Ancestry’s data team raised it from the default 5 to 100 to compact a 100-billion-row Iceberg table, because the default couldn’t keep up with their update frequency.

The follow-ups people forget. Compaction creates a new snapshot but leaves the old files on disk — old snapshots pin them. Run the cleanup chain, in this order:

-- 1. Expire old snapshots so their files can actually be deleted
--    (defaults: older_than = 5 days ago, retain_last = 1)
CALL lakehouse.system.expire_snapshots(
  table       => 'order_events',
  older_than  => TIMESTAMP '2026-10-01 00:00:00',
  retain_last => 10
);

-- 2. Consolidate manifest files (metadata-only, fast)
CALL lakehouse.system.rewrite_manifests(table => 'order_events');

-- 3. Sweep up orphan files from failed/partial writes (default window: 3 days)
--    Use dry_run first, and never set older_than shorter than your longest write.
CALL lakehouse.system.remove_orphan_files(
  table      => 'order_events',
  older_than => TIMESTAMP '2026-10-10 00:00:00',
  dry_run    => true
);

Compacting without ever expiring snapshots means storage grows forever. And remove_orphan_files with too aggressive a window can delete files from in-flight writes — multi-day windows are normal; hours are not.

Step 5: Keep It Healthy — the Index-Card Checklist

The industry rule of thumb is 128–512 MB per file for analytical workloads, and Iceberg’s own defaults agree: write.target-file-size-bytes is 512 MB, and the read path combines splits toward 128 MB (read.split.target-size).

Five lines, taped to your monitor:

  1. Measure first. Query tbl.files monthly (weekly for streaming tables). Alert when average file size drops below ~128 MB or file count crosses your threshold.
  2. Fix the writer. Set write.target-file-size-bytes, use distribution-mode=hash and fanout-enabled=true on streaming writes, and size your trigger intervals to the data volume.
  3. Compact on a cadence that matches ingestion. Streaming tables often need compaction every few hours, not nightly; batch tables can go weekly.
  4. Always expire snapshots and rewrite manifests after compacting. Compaction without cleanup is half a job.
  5. Sweep orphans quarterly with a safe older_than window — on mature streaming lakes, orphans routinely account for a quarter or more of billable storage (source).

Variations Worth Knowing

Delete-heavy tables (merge-on-read)?

Watch delete-file counts in tbl.files too — compact position deletes with rewrite_position_delete_files, not just data files.

Sub-hour freshness needs?

Don’t chase 512 MB if it means holding micro-batches open. 128–256 MB is fine for streaming tables where freshness matters more than scan efficiency.

Heavy point lookups?

Smaller files with tight clustering (sort/Z-order) beat larger unsorted ones. File size is a means to an end — the end is pruning.

✅ Conclusion

Small files are not a moral failing — they’re the natural output of a healthy streaming pipeline. The failure is letting the write path run unsupervised. Measure the file layout like you’d measure query latency, and the problem stays a maintenance task instead of becoming an incident.

If your Kafka-to-Iceberg pipeline is past the demo stage, its next chapter isn’t a new feature — it’s maintenance. Measure tbl.files, fix the writer, compact on cadence.

💬 Let’s Discuss

Are you compacting on a schedule, or only when queries get slow? Binpack or sort — which strategy do you reach for first? And what’s the worst small-files incident you’ve seen in production? Share your setup or war stories in the comments!

Additional Details

  • Description: A field guide to diagnosing, preventing, and fixing the small-files problem in Spark + Apache Iceberg lakehouse tables — with real numbers, working SQL, and a maintenance checklist.
11 min read

MCP for Data Engineers: Connecting AI Agents to Your Iceberg Lakehouse

Your AI Agent Can’t See Your Lakehouse - Let’s Fix That

How many times this year have you pasted a table schema into a chat window so the AI could help you write a query?

In my last post, we streamed Kafka events into an Apache Iceberg lakehouse with Spark Structured Streaming. The natural follow-up in 2026: instead of querying that table myself, can I just ask an AI agent what’s in it - and get a real answer from the live catalog, not a pasted screenshot?

That is exactly what the Model Context Protocol (MCP) is for. In this post, we’ll build a small MCP server that sits on top of an Iceberg catalog and exposes it to any MCP-capable agent (Claude, IDEs, your own agents):

  • What MCP actually is, in data-engineering terms
  • A working MCP server over an Iceberg catalog (Python, ~60 lines)
  • Wiring it into an agent host like Claude
  • The security lines you should not cross

Every code sample below was run end to end while writing this post - client, server, and a real Iceberg table.

8 min read

Streaming from Kafka into an Iceberg Lakehouse, End to End

From Kafka Topics to a Lakehouse Table, the Right Way

Still landing Kafka data into HDFS or S3 as raw files and batching it later?

Back in 2015, I wrote about moving Twitter streams from Kafka to HDFS with Camus. A decade later, the goal is the same - get streaming data durably into storage - but the destination has changed completely. Today, we write straight into an Apache Iceberg table and get ACID commits, schema evolution, and time travel for free.

In this post, we’ll build the whole pipeline end to end:

  • A Kafka topic producing JSON events
  • A Spark Structured Streaming job that consumes the topic
  • Streaming writes into an Iceberg table with exactly-once semantics
  • Querying and maintaining the table (compaction, snapshots)
5 min read

Unlocking Data Lakes with Apache Iceberg: The Open Table Format Revolutionizing Analytics

Why Apache Iceberg is the Game-Changer Your Data Lake Needs

Still relying on traditional data lakes with unreliable reads, lack of ACID guarantees, and painful schema evolution?

Welcome to Apache Iceberg—an open table format purpose-built for modern, petabyte-scale analytics. With native support for engines like Apache Spark, Trino, Presto, Hive, and Flink, Iceberg solves the biggest challenges in big data processing.

In this post, we’ll break down:

  • What Apache Iceberg is and why it matters
  • How its architecture works (with a visual!)
  • Key features that make it future-ready
  • Real-world use cases
  • A call to action for your next data platform decision
3 min read
Back to Top ↑

apache-spark

What does Skipped Stage means in Spark WebUI ?

Skipped Stages in Spark UI

You must have come across various scenarios where you see a DAG like below, where you see a few stages shows greyed out with a text (skipped) after the stage name. What does this mean? Did Spark ignore one of your stage due to an error? or this is due to something else? Well, it’s actually a good thing. It means that particular stage in the lineage DAG doesn’t need to be re-evaluated as its already evaluated and cached. This will save computation time for that stage. If you want to see what data frame for that stage was stored in the cache, you can check in Storage tab in Spark UI.

~1 min read

Dataframe Operations in Spark using Scala

A comprehensive, developer-friendly guide to common DataFrame operations in Apache Spark using Scala, with code examples and explanations for each join type.

2 min read
Back to Top ↑

lakehouse

The Small-Files Problem: A Field Guide

The Maintenance Chapter Your Streaming Pipeline Was Missing

Your streaming job writes 2 MB Parquet files every two minutes. Six months later, your dashboard queries are ten times slower — and nobody changed the data. Sound familiar?

Yesterday, in Streaming from Kafka into an Iceberg Lakehouse, End to End, we built the whole pipeline — Kafka topic, Spark Structured Streaming, Iceberg table — and Step 6 ended with a warning: one-minute micro-batches create lots of small files — the classic streaming lakehouse problem. This post picks up exactly there. The pipeline works; the file layout is what’s rotting underneath it.

Same stack as yesterday: Kafka 4.3.1, Spark 4.2.0 (Scala 2.13), Iceberg 1.12.0, PySpark — and the same lakehouse.order_events table, partitioned by days(event_time).

In this post, we’ll:

  • See why small files hurt — with real numbers, not vibes

  • Diagnose the problem using Iceberg’s own metadata tables

  • Stop writing small files at write time

  • Fix the damage with compaction (rewrite_data_files, done right)

  • Keep the layout healthy with a checklist that fits on an index card

Step 1: Understand Why Small Files Hurt

It’s tempting to treat file count as cosmetic. It isn’t. Every file in your table is a tax paid four times: at plan time, at task-scheduling time, at request time, and at footer-read time.

Query planning. Iceberg query planning reads manifest files to decide which data files to scan — the manifest and snapshot layer I broke down in the Iceberg intro is exactly what gets stressed here. Dremio’s engineers measured this directly: the same dataset stored in 2,000 files of 256 MB each typically plans in under one second, while a table fragmented into 100,000 files pays a planning cost an order of magnitude higher. Manifests fragment too — every micro-batch commit adds manifests, and the metadata tree stops being an index and becomes a maze.

Task scheduling. Each file becomes at least one execution task. Scheduling costs roughly 1–10 milliseconds per task — trivial, until you multiply: a query against 100,000 files creates at least 100,000 tasks. Even if partition pruning eliminates 95% of them, the surviving 5,000 tasks carry 5–50 seconds of pure scheduling overhead before any data moves. On JVM engines like Spark, giant task graphs also pile on GC pressure, adding unpredictable latency.

Object-store requests. S3 charges $0.0004 per 1,000 GET requests. Reading a file takes at minimum one GET, but Parquet readers also fetch the footer and individual row groups — conservatively, two requests per file. So 100,000 files means 200,000 requests, or $0.08 per query, in storage API costs alone. A table queried 10,000 times a day burns $800 a day — $24,000 a month — on requests. Compact to 2,000 files and that drops to about $1,600 a month, a 93% reduction (source).

Footer reads. Every Parquet file carries a footer — schema, row-group metadata, column statistics. Reading it is mandatory and costs roughly the same whether the file is 1 MB or 512 MB: one to several milliseconds plus a network round trip. For a 256 MB file that’s noise; for a 1 MB file it dominates. If 95% of your read time goes to footers instead of column data, you have a severe small-files problem (source).

How fast it accumulates. Streaming engines checkpoint every 1–10 minutes, and each checkpoint commits new Parquet files — one per partition. A table with 100 partitions on 10-minute checkpoints produces 14,400 new files per day, most of them a few megabytes each (source). A table with 500,000 small files queried 50 times a day generates roughly 25 million GET requests a month. The write path that felt so elegant on day one is a liability by month six.

Step 2: Diagnose — Is This Actually Your Problem?

Don’t compact on vibes. Iceberg gives you metadata tables that answer the question directly:

-- File health at a glance: count, average size, smallest/largest
SELECT
  COUNT(*)                                        AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)     AS avg_mb,
  ROUND(MIN(file_size_in_bytes) / 1048576, 2)     AS min_mb,
  ROUND(MAX(file_size_in_bytes) / 1048576, 2)     AS max_mb,
  SUM(CASE WHEN file_size_in_bytes < 134217728
           THEN 1 ELSE 0 END)                     AS files_under_128mb
FROM lakehouse.order_events.files
WHERE content = 0;   -- 0 = DATA files; 1/2 = delete files

If a large share of files sits under 128 MB — or you see thousands of files averaging a few MB — you have the problem. Next, find where it hurts most, so your first compaction pass buys the biggest win:

-- Worst partitions first: file count and avg size per partition
SELECT
  partition,
  COUNT(*)                                     AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)  AS avg_mb
FROM lakehouse.order_events.files
WHERE content = 0
GROUP BY partition
ORDER BY file_count DESC
LIMIT 30;

Two more signals worth checking:

-- Are snapshots piling up from streaming appends?
SELECT snapshot_id, committed_at, operation, summary
FROM lakehouse.order_events.snapshots
ORDER BY committed_at DESC
LIMIT 20;

A wall of append snapshots with no replace (compaction) in sight confirms the diagnosis. And don’t forget the manifests metadata table — if manifest count is climbing with file count, planning overhead is compounding.

Outside SQL, the symptoms are just as telling: Spark UI jobs where a stage fans out into tens of thousands of tiny tasks, query plans that take longer than the scan itself, and — the one people notice last — your S3 GetObject/ListObjects request metrics climbing month over month while data volume stays flat.

Step 3: Fix It at Write Time

Compaction is corrective. The cheaper move is to stop writing small files in the first place. Three things worth knowing about the write path:

1. Target file size. The table property write.target-file-size-bytes defaults to 512 MB. Spark writers roll a new file when buffered data crosses this threshold — but only if a single task actually accumulates that much. A streaming micro-batch with 50 MB of data never reaches it, which is exactly how you get 2 MB files:

-- Set once at the table level (256 MB is a common streaming-table choice)
ALTER TABLE lakehouse.order_events
  SET TBLPROPERTIES ('write.target-file-size-bytes' = '268435456');

You can also override it per write with the target-file-size-bytes DataFrame option.

2. Distribution mode. write.distribution-mode decides how rows are shuffled before writing: none (no shuffle — each task writes whatever it has; the small-files factory), hash (hash-distribute by partition key), or range (range-distribute by partition or sort key). For streaming writes into a partitioned table, hash is usually the right call:

(df.writeStream
   .format("iceberg")
   .option("checkpointLocation", checkpoint_path)
   .option("distribution-mode", "hash")   # shuffle by partition key before writing
   .option("fanout-enabled", "true")      # keep partition writers open across batches
   .toTable("lakehouse.order_events"))

3. The fanout writer. Streaming data arrives out of order across partitions. Without fanout, every partition switch inside a task closes the current file and opens a new one — file rollover churn. write.spark.fanout.enabled (default false) keeps multiple partition writers open simultaneously, at the cost of more memory. For streaming jobs, enable it.

We used fanout-enabled: true in the streaming job from yesterday’s post for exactly this reason — without it, out-of-order events across days would have multiplied our file count from the very first batch.

Two more write-path habits that pay off:

  • repartition before batch writes. Line writers up with the partition spec so each task writes fewer, larger files:
from pyspark.sql.functions import to_date

def write_batch(batch_df, batch_id):
    (batch_df
       .repartition(to_date("event_time"))  # align writers with days(event_time)
       .writeTo("lakehouse.order_events")
       .option("fanout-enabled", "true")
       .append())

(query.writeStream
   .foreachBatch(write_batch)
   .option("checkpointLocation", checkpoint_path)
   .trigger(availableNow=True)
   .start())
  • Trigger intervals. The one-minute trigger we used yesterday is great for freshness demos and terrible for file layout — on a low-volume topic it guarantees tiny files. Prefer Trigger.AvailableNow or multi-minute processing-time triggers, and coalesce inside the batch function as above.

None of this eliminates compaction entirely — streaming tables will always drift — but it slows the drift from days to weeks.

Step 4: Compact What’s Already There

Yesterday’s post showed the one-line version — CALL lakehouse.system.rewrite_data_files(table => 'order_events'). Here’s the full picture. Iceberg’s compaction primitive rewrites selected data files into fewer, larger ones and commits the result as a new snapshot — fully online, no downtime; readers keep working against the old snapshot until the new one commits.

Binpack (default): fast and dumb, in a good way. It groups small files by partition and concatenates them up to the target size. No shuffle, no reordering — the cheapest possible rewrite:

-- Compact only the files that need it: small files in recent partitions
CALL lakehouse.system.rewrite_data_files(
  table    => 'order_events',
  strategy => 'binpack',
  where    => 'event_time >= current_timestamp() - INTERVAL 7 DAYS',
  options  => map(
    'target-file-size-bytes', '268435456',   -- 256 MB
    'min-input-files',        '5'             -- skip groups already healthy
  )
);

Sort: slower, but fixes layout too. Binpack doesn’t touch data ordering. If your queries filter on event_time and customer_id but files were written in arrival order, column statistics are weak and pruning suffers. The sort strategy globally sorts before allocating files to tasks, tightening min/max ranges per file:

-- Merge + cluster by the columns your queries actually filter on
CALL lakehouse.system.rewrite_data_files(
  table      => 'order_events',
  strategy   => 'sort',
  sort_order => 'event_time ASC NULLS LAST, customer_id ASC NULLS LAST',
  options    => map(
    'target-file-size-bytes',             '268435456',
    'partial-progress.enabled',           'true',
    'max-concurrent-file-group-rewrites', '5'
  )
);

-- For multi-column filters with no single dominant column, Z-order instead:
-- sort_order => 'zorder(event_time, customer_id)'

(Iceberg 1.12 also adds Hilbert-curve clustering as a newer alternative to Z-ordering — worth evaluating once your engine supports it.)

The options that matter in production:

Option Default What it does
target-file-size-bytes table’s write.target-file-size-bytes (512 MB) Output file size target
min-input-files 5 File groups with fewer files are skipped
min-file-size-bytes 75% of target Files below this are always candidates
max-file-size-bytes 180% of target Files above this are always candidates
max-concurrent-file-group-rewrites 5 Parallelism of the rewrite; raise for large tables
max-file-group-size-bytes 100 GB Splits the job into file groups
partial-progress.enabled false Commit file groups as they finish instead of one big commit
partial-progress.max-commits 10 Cap on commits when partial progress is on
rewrite-all false Force-rewrite everything, ignoring the filters above
rewrite-job-order none Order groups: bytes-asc, bytes-desc, files-asc, files-desc

Two of these deserve emphasis. partial-progress.enabled=true is the difference between losing hours of rewrite work on a mid-job failure and keeping what finished — always turn it on for large tables. And max-concurrent-file-group-rewrites is the knob real teams reach for at scale: Ancestry’s data team raised it from the default 5 to 100 to compact a 100-billion-row Iceberg table, because the default couldn’t keep up with their update frequency.

The follow-ups people forget. Compaction creates a new snapshot but leaves the old files on disk — old snapshots pin them. Run the cleanup chain, in this order:

-- 1. Expire old snapshots so their files can actually be deleted
--    (defaults: older_than = 5 days ago, retain_last = 1)
CALL lakehouse.system.expire_snapshots(
  table       => 'order_events',
  older_than  => TIMESTAMP '2026-10-01 00:00:00',
  retain_last => 10
);

-- 2. Consolidate manifest files (metadata-only, fast)
CALL lakehouse.system.rewrite_manifests(table => 'order_events');

-- 3. Sweep up orphan files from failed/partial writes (default window: 3 days)
--    Use dry_run first, and never set older_than shorter than your longest write.
CALL lakehouse.system.remove_orphan_files(
  table      => 'order_events',
  older_than => TIMESTAMP '2026-10-10 00:00:00',
  dry_run    => true
);

Compacting without ever expiring snapshots means storage grows forever. And remove_orphan_files with too aggressive a window can delete files from in-flight writes — multi-day windows are normal; hours are not.

Step 5: Keep It Healthy — the Index-Card Checklist

The industry rule of thumb is 128–512 MB per file for analytical workloads, and Iceberg’s own defaults agree: write.target-file-size-bytes is 512 MB, and the read path combines splits toward 128 MB (read.split.target-size).

Five lines, taped to your monitor:

  1. Measure first. Query tbl.files monthly (weekly for streaming tables). Alert when average file size drops below ~128 MB or file count crosses your threshold.
  2. Fix the writer. Set write.target-file-size-bytes, use distribution-mode=hash and fanout-enabled=true on streaming writes, and size your trigger intervals to the data volume.
  3. Compact on a cadence that matches ingestion. Streaming tables often need compaction every few hours, not nightly; batch tables can go weekly.
  4. Always expire snapshots and rewrite manifests after compacting. Compaction without cleanup is half a job.
  5. Sweep orphans quarterly with a safe older_than window — on mature streaming lakes, orphans routinely account for a quarter or more of billable storage (source).

Variations Worth Knowing

Delete-heavy tables (merge-on-read)?

Watch delete-file counts in tbl.files too — compact position deletes with rewrite_position_delete_files, not just data files.

Sub-hour freshness needs?

Don’t chase 512 MB if it means holding micro-batches open. 128–256 MB is fine for streaming tables where freshness matters more than scan efficiency.

Heavy point lookups?

Smaller files with tight clustering (sort/Z-order) beat larger unsorted ones. File size is a means to an end — the end is pruning.

✅ Conclusion

Small files are not a moral failing — they’re the natural output of a healthy streaming pipeline. The failure is letting the write path run unsupervised. Measure the file layout like you’d measure query latency, and the problem stays a maintenance task instead of becoming an incident.

If your Kafka-to-Iceberg pipeline is past the demo stage, its next chapter isn’t a new feature — it’s maintenance. Measure tbl.files, fix the writer, compact on cadence.

💬 Let’s Discuss

Are you compacting on a schedule, or only when queries get slow? Binpack or sort — which strategy do you reach for first? And what’s the worst small-files incident you’ve seen in production? Share your setup or war stories in the comments!

Additional Details

  • Description: A field guide to diagnosing, preventing, and fixing the small-files problem in Spark + Apache Iceberg lakehouse tables — with real numbers, working SQL, and a maintenance checklist.
11 min read

MCP for Data Engineers: Connecting AI Agents to Your Iceberg Lakehouse

Your AI Agent Can’t See Your Lakehouse - Let’s Fix That

How many times this year have you pasted a table schema into a chat window so the AI could help you write a query?

In my last post, we streamed Kafka events into an Apache Iceberg lakehouse with Spark Structured Streaming. The natural follow-up in 2026: instead of querying that table myself, can I just ask an AI agent what’s in it - and get a real answer from the live catalog, not a pasted screenshot?

That is exactly what the Model Context Protocol (MCP) is for. In this post, we’ll build a small MCP server that sits on top of an Iceberg catalog and exposes it to any MCP-capable agent (Claude, IDEs, your own agents):

  • What MCP actually is, in data-engineering terms
  • A working MCP server over an Iceberg catalog (Python, ~60 lines)
  • Wiring it into an agent host like Claude
  • The security lines you should not cross

Every code sample below was run end to end while writing this post - client, server, and a real Iceberg table.

8 min read

Streaming from Kafka into an Iceberg Lakehouse, End to End

From Kafka Topics to a Lakehouse Table, the Right Way

Still landing Kafka data into HDFS or S3 as raw files and batching it later?

Back in 2015, I wrote about moving Twitter streams from Kafka to HDFS with Camus. A decade later, the goal is the same - get streaming data durably into storage - but the destination has changed completely. Today, we write straight into an Apache Iceberg table and get ACID commits, schema evolution, and time travel for free.

In this post, we’ll build the whole pipeline end to end:

  • A Kafka topic producing JSON events
  • A Spark Structured Streaming job that consumes the topic
  • Streaming writes into an Iceberg table with exactly-once semantics
  • Querying and maintaining the table (compaction, snapshots)
5 min read
Back to Top ↑

data engineering

The Small-Files Problem: A Field Guide

The Maintenance Chapter Your Streaming Pipeline Was Missing

Your streaming job writes 2 MB Parquet files every two minutes. Six months later, your dashboard queries are ten times slower — and nobody changed the data. Sound familiar?

Yesterday, in Streaming from Kafka into an Iceberg Lakehouse, End to End, we built the whole pipeline — Kafka topic, Spark Structured Streaming, Iceberg table — and Step 6 ended with a warning: one-minute micro-batches create lots of small files — the classic streaming lakehouse problem. This post picks up exactly there. The pipeline works; the file layout is what’s rotting underneath it.

Same stack as yesterday: Kafka 4.3.1, Spark 4.2.0 (Scala 2.13), Iceberg 1.12.0, PySpark — and the same lakehouse.order_events table, partitioned by days(event_time).

In this post, we’ll:

  • See why small files hurt — with real numbers, not vibes

  • Diagnose the problem using Iceberg’s own metadata tables

  • Stop writing small files at write time

  • Fix the damage with compaction (rewrite_data_files, done right)

  • Keep the layout healthy with a checklist that fits on an index card

Step 1: Understand Why Small Files Hurt

It’s tempting to treat file count as cosmetic. It isn’t. Every file in your table is a tax paid four times: at plan time, at task-scheduling time, at request time, and at footer-read time.

Query planning. Iceberg query planning reads manifest files to decide which data files to scan — the manifest and snapshot layer I broke down in the Iceberg intro is exactly what gets stressed here. Dremio’s engineers measured this directly: the same dataset stored in 2,000 files of 256 MB each typically plans in under one second, while a table fragmented into 100,000 files pays a planning cost an order of magnitude higher. Manifests fragment too — every micro-batch commit adds manifests, and the metadata tree stops being an index and becomes a maze.

Task scheduling. Each file becomes at least one execution task. Scheduling costs roughly 1–10 milliseconds per task — trivial, until you multiply: a query against 100,000 files creates at least 100,000 tasks. Even if partition pruning eliminates 95% of them, the surviving 5,000 tasks carry 5–50 seconds of pure scheduling overhead before any data moves. On JVM engines like Spark, giant task graphs also pile on GC pressure, adding unpredictable latency.

Object-store requests. S3 charges $0.0004 per 1,000 GET requests. Reading a file takes at minimum one GET, but Parquet readers also fetch the footer and individual row groups — conservatively, two requests per file. So 100,000 files means 200,000 requests, or $0.08 per query, in storage API costs alone. A table queried 10,000 times a day burns $800 a day — $24,000 a month — on requests. Compact to 2,000 files and that drops to about $1,600 a month, a 93% reduction (source).

Footer reads. Every Parquet file carries a footer — schema, row-group metadata, column statistics. Reading it is mandatory and costs roughly the same whether the file is 1 MB or 512 MB: one to several milliseconds plus a network round trip. For a 256 MB file that’s noise; for a 1 MB file it dominates. If 95% of your read time goes to footers instead of column data, you have a severe small-files problem (source).

How fast it accumulates. Streaming engines checkpoint every 1–10 minutes, and each checkpoint commits new Parquet files — one per partition. A table with 100 partitions on 10-minute checkpoints produces 14,400 new files per day, most of them a few megabytes each (source). A table with 500,000 small files queried 50 times a day generates roughly 25 million GET requests a month. The write path that felt so elegant on day one is a liability by month six.

Step 2: Diagnose — Is This Actually Your Problem?

Don’t compact on vibes. Iceberg gives you metadata tables that answer the question directly:

-- File health at a glance: count, average size, smallest/largest
SELECT
  COUNT(*)                                        AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)     AS avg_mb,
  ROUND(MIN(file_size_in_bytes) / 1048576, 2)     AS min_mb,
  ROUND(MAX(file_size_in_bytes) / 1048576, 2)     AS max_mb,
  SUM(CASE WHEN file_size_in_bytes < 134217728
           THEN 1 ELSE 0 END)                     AS files_under_128mb
FROM lakehouse.order_events.files
WHERE content = 0;   -- 0 = DATA files; 1/2 = delete files

If a large share of files sits under 128 MB — or you see thousands of files averaging a few MB — you have the problem. Next, find where it hurts most, so your first compaction pass buys the biggest win:

-- Worst partitions first: file count and avg size per partition
SELECT
  partition,
  COUNT(*)                                     AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)  AS avg_mb
FROM lakehouse.order_events.files
WHERE content = 0
GROUP BY partition
ORDER BY file_count DESC
LIMIT 30;

Two more signals worth checking:

-- Are snapshots piling up from streaming appends?
SELECT snapshot_id, committed_at, operation, summary
FROM lakehouse.order_events.snapshots
ORDER BY committed_at DESC
LIMIT 20;

A wall of append snapshots with no replace (compaction) in sight confirms the diagnosis. And don’t forget the manifests metadata table — if manifest count is climbing with file count, planning overhead is compounding.

Outside SQL, the symptoms are just as telling: Spark UI jobs where a stage fans out into tens of thousands of tiny tasks, query plans that take longer than the scan itself, and — the one people notice last — your S3 GetObject/ListObjects request metrics climbing month over month while data volume stays flat.

Step 3: Fix It at Write Time

Compaction is corrective. The cheaper move is to stop writing small files in the first place. Three things worth knowing about the write path:

1. Target file size. The table property write.target-file-size-bytes defaults to 512 MB. Spark writers roll a new file when buffered data crosses this threshold — but only if a single task actually accumulates that much. A streaming micro-batch with 50 MB of data never reaches it, which is exactly how you get 2 MB files:

-- Set once at the table level (256 MB is a common streaming-table choice)
ALTER TABLE lakehouse.order_events
  SET TBLPROPERTIES ('write.target-file-size-bytes' = '268435456');

You can also override it per write with the target-file-size-bytes DataFrame option.

2. Distribution mode. write.distribution-mode decides how rows are shuffled before writing: none (no shuffle — each task writes whatever it has; the small-files factory), hash (hash-distribute by partition key), or range (range-distribute by partition or sort key). For streaming writes into a partitioned table, hash is usually the right call:

(df.writeStream
   .format("iceberg")
   .option("checkpointLocation", checkpoint_path)
   .option("distribution-mode", "hash")   # shuffle by partition key before writing
   .option("fanout-enabled", "true")      # keep partition writers open across batches
   .toTable("lakehouse.order_events"))

3. The fanout writer. Streaming data arrives out of order across partitions. Without fanout, every partition switch inside a task closes the current file and opens a new one — file rollover churn. write.spark.fanout.enabled (default false) keeps multiple partition writers open simultaneously, at the cost of more memory. For streaming jobs, enable it.

We used fanout-enabled: true in the streaming job from yesterday’s post for exactly this reason — without it, out-of-order events across days would have multiplied our file count from the very first batch.

Two more write-path habits that pay off:

  • repartition before batch writes. Line writers up with the partition spec so each task writes fewer, larger files:
from pyspark.sql.functions import to_date

def write_batch(batch_df, batch_id):
    (batch_df
       .repartition(to_date("event_time"))  # align writers with days(event_time)
       .writeTo("lakehouse.order_events")
       .option("fanout-enabled", "true")
       .append())

(query.writeStream
   .foreachBatch(write_batch)
   .option("checkpointLocation", checkpoint_path)
   .trigger(availableNow=True)
   .start())
  • Trigger intervals. The one-minute trigger we used yesterday is great for freshness demos and terrible for file layout — on a low-volume topic it guarantees tiny files. Prefer Trigger.AvailableNow or multi-minute processing-time triggers, and coalesce inside the batch function as above.

None of this eliminates compaction entirely — streaming tables will always drift — but it slows the drift from days to weeks.

Step 4: Compact What’s Already There

Yesterday’s post showed the one-line version — CALL lakehouse.system.rewrite_data_files(table => 'order_events'). Here’s the full picture. Iceberg’s compaction primitive rewrites selected data files into fewer, larger ones and commits the result as a new snapshot — fully online, no downtime; readers keep working against the old snapshot until the new one commits.

Binpack (default): fast and dumb, in a good way. It groups small files by partition and concatenates them up to the target size. No shuffle, no reordering — the cheapest possible rewrite:

-- Compact only the files that need it: small files in recent partitions
CALL lakehouse.system.rewrite_data_files(
  table    => 'order_events',
  strategy => 'binpack',
  where    => 'event_time >= current_timestamp() - INTERVAL 7 DAYS',
  options  => map(
    'target-file-size-bytes', '268435456',   -- 256 MB
    'min-input-files',        '5'             -- skip groups already healthy
  )
);

Sort: slower, but fixes layout too. Binpack doesn’t touch data ordering. If your queries filter on event_time and customer_id but files were written in arrival order, column statistics are weak and pruning suffers. The sort strategy globally sorts before allocating files to tasks, tightening min/max ranges per file:

-- Merge + cluster by the columns your queries actually filter on
CALL lakehouse.system.rewrite_data_files(
  table      => 'order_events',
  strategy   => 'sort',
  sort_order => 'event_time ASC NULLS LAST, customer_id ASC NULLS LAST',
  options    => map(
    'target-file-size-bytes',             '268435456',
    'partial-progress.enabled',           'true',
    'max-concurrent-file-group-rewrites', '5'
  )
);

-- For multi-column filters with no single dominant column, Z-order instead:
-- sort_order => 'zorder(event_time, customer_id)'

(Iceberg 1.12 also adds Hilbert-curve clustering as a newer alternative to Z-ordering — worth evaluating once your engine supports it.)

The options that matter in production:

Option Default What it does
target-file-size-bytes table’s write.target-file-size-bytes (512 MB) Output file size target
min-input-files 5 File groups with fewer files are skipped
min-file-size-bytes 75% of target Files below this are always candidates
max-file-size-bytes 180% of target Files above this are always candidates
max-concurrent-file-group-rewrites 5 Parallelism of the rewrite; raise for large tables
max-file-group-size-bytes 100 GB Splits the job into file groups
partial-progress.enabled false Commit file groups as they finish instead of one big commit
partial-progress.max-commits 10 Cap on commits when partial progress is on
rewrite-all false Force-rewrite everything, ignoring the filters above
rewrite-job-order none Order groups: bytes-asc, bytes-desc, files-asc, files-desc

Two of these deserve emphasis. partial-progress.enabled=true is the difference between losing hours of rewrite work on a mid-job failure and keeping what finished — always turn it on for large tables. And max-concurrent-file-group-rewrites is the knob real teams reach for at scale: Ancestry’s data team raised it from the default 5 to 100 to compact a 100-billion-row Iceberg table, because the default couldn’t keep up with their update frequency.

The follow-ups people forget. Compaction creates a new snapshot but leaves the old files on disk — old snapshots pin them. Run the cleanup chain, in this order:

-- 1. Expire old snapshots so their files can actually be deleted
--    (defaults: older_than = 5 days ago, retain_last = 1)
CALL lakehouse.system.expire_snapshots(
  table       => 'order_events',
  older_than  => TIMESTAMP '2026-10-01 00:00:00',
  retain_last => 10
);

-- 2. Consolidate manifest files (metadata-only, fast)
CALL lakehouse.system.rewrite_manifests(table => 'order_events');

-- 3. Sweep up orphan files from failed/partial writes (default window: 3 days)
--    Use dry_run first, and never set older_than shorter than your longest write.
CALL lakehouse.system.remove_orphan_files(
  table      => 'order_events',
  older_than => TIMESTAMP '2026-10-10 00:00:00',
  dry_run    => true
);

Compacting without ever expiring snapshots means storage grows forever. And remove_orphan_files with too aggressive a window can delete files from in-flight writes — multi-day windows are normal; hours are not.

Step 5: Keep It Healthy — the Index-Card Checklist

The industry rule of thumb is 128–512 MB per file for analytical workloads, and Iceberg’s own defaults agree: write.target-file-size-bytes is 512 MB, and the read path combines splits toward 128 MB (read.split.target-size).

Five lines, taped to your monitor:

  1. Measure first. Query tbl.files monthly (weekly for streaming tables). Alert when average file size drops below ~128 MB or file count crosses your threshold.
  2. Fix the writer. Set write.target-file-size-bytes, use distribution-mode=hash and fanout-enabled=true on streaming writes, and size your trigger intervals to the data volume.
  3. Compact on a cadence that matches ingestion. Streaming tables often need compaction every few hours, not nightly; batch tables can go weekly.
  4. Always expire snapshots and rewrite manifests after compacting. Compaction without cleanup is half a job.
  5. Sweep orphans quarterly with a safe older_than window — on mature streaming lakes, orphans routinely account for a quarter or more of billable storage (source).

Variations Worth Knowing

Delete-heavy tables (merge-on-read)?

Watch delete-file counts in tbl.files too — compact position deletes with rewrite_position_delete_files, not just data files.

Sub-hour freshness needs?

Don’t chase 512 MB if it means holding micro-batches open. 128–256 MB is fine for streaming tables where freshness matters more than scan efficiency.

Heavy point lookups?

Smaller files with tight clustering (sort/Z-order) beat larger unsorted ones. File size is a means to an end — the end is pruning.

✅ Conclusion

Small files are not a moral failing — they’re the natural output of a healthy streaming pipeline. The failure is letting the write path run unsupervised. Measure the file layout like you’d measure query latency, and the problem stays a maintenance task instead of becoming an incident.

If your Kafka-to-Iceberg pipeline is past the demo stage, its next chapter isn’t a new feature — it’s maintenance. Measure tbl.files, fix the writer, compact on cadence.

💬 Let’s Discuss

Are you compacting on a schedule, or only when queries get slow? Binpack or sort — which strategy do you reach for first? And what’s the worst small-files incident you’ve seen in production? Share your setup or war stories in the comments!

Additional Details

  • Description: A field guide to diagnosing, preventing, and fixing the small-files problem in Spark + Apache Iceberg lakehouse tables — with real numbers, working SQL, and a maintenance checklist.
11 min read

MCP for Data Engineers: Connecting AI Agents to Your Iceberg Lakehouse

Your AI Agent Can’t See Your Lakehouse - Let’s Fix That

How many times this year have you pasted a table schema into a chat window so the AI could help you write a query?

In my last post, we streamed Kafka events into an Apache Iceberg lakehouse with Spark Structured Streaming. The natural follow-up in 2026: instead of querying that table myself, can I just ask an AI agent what’s in it - and get a real answer from the live catalog, not a pasted screenshot?

That is exactly what the Model Context Protocol (MCP) is for. In this post, we’ll build a small MCP server that sits on top of an Iceberg catalog and exposes it to any MCP-capable agent (Claude, IDEs, your own agents):

  • What MCP actually is, in data-engineering terms
  • A working MCP server over an Iceberg catalog (Python, ~60 lines)
  • Wiring it into an agent host like Claude
  • The security lines you should not cross

Every code sample below was run end to end while writing this post - client, server, and a real Iceberg table.

8 min read

Streaming from Kafka into an Iceberg Lakehouse, End to End

From Kafka Topics to a Lakehouse Table, the Right Way

Still landing Kafka data into HDFS or S3 as raw files and batching it later?

Back in 2015, I wrote about moving Twitter streams from Kafka to HDFS with Camus. A decade later, the goal is the same - get streaming data durably into storage - but the destination has changed completely. Today, we write straight into an Apache Iceberg table and get ACID commits, schema evolution, and time travel for free.

In this post, we’ll build the whole pipeline end to end:

  • A Kafka topic producing JSON events
  • A Spark Structured Streaming job that consumes the topic
  • Streaming writes into an Iceberg table with exactly-once semantics
  • Querying and maintaining the table (compaction, snapshots)
5 min read
Back to Top ↑

Java Interview

Java : What does finalize do and How?

finalize method in the Object class is often a point of discussion regarding whether it should be used or not. Below are some important pointers on the finalize method:

2 min read
Back to Top ↑

Pig

Hadoop : Getting Started with Pig

What is Apache Pig?

Apache Pig is a high level scripting language that is used with Apache Hadoop. It enables data analysts to write complex data transformations without knowing Java. Its simple SQL-like scripting language is called Pig Latin, and appeals to developers already familiar with scripting languages and SQL. Pig Scripts are converted into MapReduce Jobs which run on data stored in HDFS (refer to the diagram below).

3 min read
Back to Top ↑

metastore

How-To : Connect HiveServer2 service with JDBC Client ?

HiveServer2 (HS2) is a server interface that enables remote clients to execute queries against Hive and retrieve the results. The current implementation, based on Thrift RPC, is an improved version of HiveServer and supports multi-client concurrency and authentication. It is designed to provide better support for open API clients like JDBC and ODBC.

3 min read

How-To : Configure MySQL Metastore for Hive ?

Hive by default comes with Derby as its metastore storage, which is suited only for testing purposes and in most of the production scenarios it is recommended to use MySQL as a metastore. This is a step by step guide on How to Configure MySQL Metastore for Hive in place of Derby Metastore (Default).

2 min read
Back to Top ↑

hbase

How-to : Write a CoProcessor in HBase

Coprocessor is a mechanism which helps to move computations closer to the data in HBase. It is like a Mapreduce framework to distribute tasks across the cluster.

2 min read

Hive Strict Mode

![Sort By vs Order By vs Group By vs Cluster By in Hive]/assets/uploads/2015/01/images.jpg)

What is Hive Strict Mode?

Hive Strict Mode (hive.mapred.mode=strict) enables Hive to restrict certain performance intensive operations. Such as:

~1 min read
Back to Top ↑

hcatalog

What is Apache HCatalog ?

A developer-friendly introduction to Apache HCatalog, its architecture, features, and how it fits into the Hadoop ecosystem.

2 min read
Back to Top ↑

kafka

Back to Top ↑

java8

Hashmap Performance Improvements in Java 8

A developer-focused look at how Java 8 improved the performance of HashMap under high-collision scenarios, with code examples and practical explanations.

2 min read
Back to Top ↑

scala

Dataframe Operations in Spark using Scala

A comprehensive, developer-friendly guide to common DataFrame operations in Apache Spark using Scala, with code examples and explanations for each join type.

2 min read
Back to Top ↑

cloud

Top 10 Hadoop Shell Commands to manage HDFS

So you already know what Hadoop is? Why it is used for? What problems you can solve with it? And you want to know how you can deal with files on HDFS? Don’t worry, you are in the right place.

2 min read
Back to Top ↑

high availability

Top 10 Hadoop Shell Commands to manage HDFS

So you already know what Hadoop is? Why it is used for? What problems you can solve with it? And you want to know how you can deal with files on HDFS? Don’t worry, you are in the right place.

2 min read
Back to Top ↑

scalable

Top 10 Hadoop Shell Commands to manage HDFS

So you already know what Hadoop is? Why it is used for? What problems you can solve with it? And you want to know how you can deal with files on HDFS? Don’t worry, you are in the right place.

2 min read
Back to Top ↑

storage

Top 10 Hadoop Shell Commands to manage HDFS

So you already know what Hadoop is? Why it is used for? What problems you can solve with it? And you want to know how you can deal with files on HDFS? Don’t worry, you are in the right place.

2 min read
Back to Top ↑

apache

Back to Top ↑

cxf

Back to Top ↑

soap

Back to Top ↑

webservice

Back to Top ↑

Intern

Back to Top ↑

String

Back to Top ↑

Technology

Back to Top ↑

Pig Latin

Hadoop : Getting Started with Pig

What is Apache Pig?

Apache Pig is a high level scripting language that is used with Apache Hadoop. It enables data analysts to write complex data transformations without knowing Java. Its simple SQL-like scripting language is called Pig Latin, and appeals to developers already familiar with scripting languages and SQL. Pig Scripts are converted into MapReduce Jobs which run on data stored in HDFS (refer to the diagram below).

3 min read
Back to Top ↑

Centralised Logging

How-To : Setup Realtime Alalytics over Logs with ELK Stack : Elasticsearch, Logstash, Kibana?

Once we know something, we find it hard to imagine what it was like not to know it.

— Chip & Dan Heath, Authors of Made to Stick, Switch

What is the ELK stack?

The ELK stack consists of open source tools ElasticSearch, Logstash, and Kibana. These three provide a fully working real-time data analytics tool for getting wonderful information sitting on your data.

3 min read
Back to Top ↑

ElasticSearch

How-To : Setup Realtime Alalytics over Logs with ELK Stack : Elasticsearch, Logstash, Kibana?

Once we know something, we find it hard to imagine what it was like not to know it.

— Chip & Dan Heath, Authors of Made to Stick, Switch

What is the ELK stack?

The ELK stack consists of open source tools ElasticSearch, Logstash, and Kibana. These three provide a fully working real-time data analytics tool for getting wonderful information sitting on your data.

3 min read
Back to Top ↑

ELK

How-To : Setup Realtime Alalytics over Logs with ELK Stack : Elasticsearch, Logstash, Kibana?

Once we know something, we find it hard to imagine what it was like not to know it.

— Chip & Dan Heath, Authors of Made to Stick, Switch

What is the ELK stack?

The ELK stack consists of open source tools ElasticSearch, Logstash, and Kibana. These three provide a fully working real-time data analytics tool for getting wonderful information sitting on your data.

3 min read
Back to Top ↑

Kibana

How-To : Setup Realtime Alalytics over Logs with ELK Stack : Elasticsearch, Logstash, Kibana?

Once we know something, we find it hard to imagine what it was like not to know it.

— Chip & Dan Heath, Authors of Made to Stick, Switch

What is the ELK stack?

The ELK stack consists of open source tools ElasticSearch, Logstash, and Kibana. These three provide a fully working real-time data analytics tool for getting wonderful information sitting on your data.

3 min read
Back to Top ↑

LogStash

How-To : Setup Realtime Alalytics over Logs with ELK Stack : Elasticsearch, Logstash, Kibana?

Once we know something, we find it hard to imagine what it was like not to know it.

— Chip & Dan Heath, Authors of Made to Stick, Switch

What is the ELK stack?

The ELK stack consists of open source tools ElasticSearch, Logstash, and Kibana. These three provide a fully working real-time data analytics tool for getting wonderful information sitting on your data.

3 min read
Back to Top ↑

Api Docs

How-To : Generate Restful API Documentation with Swagger ?

Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services. The goal of Swagger is to enable client and documentation systems to update at the same pace as the server. The documentation of methods, parameters, and models are tightly integrated into the server code, allowing APIs to always stay in sync.

2 min read
Back to Top ↑

documentation

How-To : Generate Restful API Documentation with Swagger ?

Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services. The goal of Swagger is to enable client and documentation systems to update at the same pace as the server. The documentation of methods, parameters, and models are tightly integrated into the server code, allowing APIs to always stay in sync.

2 min read
Back to Top ↑

REST API

How-To : Generate Restful API Documentation with Swagger ?

Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services. The goal of Swagger is to enable client and documentation systems to update at the same pace as the server. The documentation of methods, parameters, and models are tightly integrated into the server code, allowing APIs to always stay in sync.

2 min read
Back to Top ↑

Spring

How-To : Generate Restful API Documentation with Swagger ?

Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services. The goal of Swagger is to enable client and documentation systems to update at the same pace as the server. The documentation of methods, parameters, and models are tightly integrated into the server code, allowing APIs to always stay in sync.

2 min read
Back to Top ↑

Spring Boot

How-To : Generate Restful API Documentation with Swagger ?

Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services. The goal of Swagger is to enable client and documentation systems to update at the same pace as the server. The documentation of methods, parameters, and models are tightly integrated into the server code, allowing APIs to always stay in sync.

2 min read
Back to Top ↑

Swagger

How-To : Generate Restful API Documentation with Swagger ?

Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services. The goal of Swagger is to enable client and documentation systems to update at the same pace as the server. The documentation of methods, parameters, and models are tightly integrated into the server code, allowing APIs to always stay in sync.

2 min read
Back to Top ↑

Finalize

Java : What does finalize do and How?

finalize method in the Object class is often a point of discussion regarding whether it should be used or not. Below are some important pointers on the finalize method:

2 min read
Back to Top ↑

Garbage collection

Java : What does finalize do and How?

finalize method in the Object class is often a point of discussion regarding whether it should be used or not. Below are some important pointers on the finalize method:

2 min read
Back to Top ↑

JVM

Java : What does finalize do and How?

finalize method in the Object class is often a point of discussion regarding whether it should be used or not. Below are some important pointers on the finalize method:

2 min read
Back to Top ↑

Poodle

Back to Top ↑

ssl

Back to Top ↑

tls

Back to Top ↑

vulnerability

Back to Top ↑

datawarehouse

How-To : Configure MySQL Metastore for Hive ?

Hive by default comes with Derby as its metastore storage, which is suited only for testing purposes and in most of the production scenarios it is recommended to use MySQL as a metastore. This is a step by step guide on How to Configure MySQL Metastore for Hive in place of Derby Metastore (Default).

2 min read
Back to Top ↑

mysql

How-To : Configure MySQL Metastore for Hive ?

Hive by default comes with Derby as its metastore storage, which is suited only for testing purposes and in most of the production scenarios it is recommended to use MySQL as a metastore. This is a step by step guide on How to Configure MySQL Metastore for Hive in place of Derby Metastore (Default).

2 min read
Back to Top ↑

hadoop interview

How-To : Connect HiveServer2 service with JDBC Client ?

HiveServer2 (HS2) is a server interface that enables remote clients to execute queries against Hive and retrieve the results. The current implementation, based on Thrift RPC, is an improved version of HiveServer and supports multi-client concurrency and authentication. It is designed to provide better support for open API clients like JDBC and ODBC.

3 min read
Back to Top ↑

hiveserver2

How-To : Connect HiveServer2 service with JDBC Client ?

HiveServer2 (HS2) is a server interface that enables remote clients to execute queries against Hive and retrieve the results. The current implementation, based on Thrift RPC, is an improved version of HiveServer and supports multi-client concurrency and authentication. It is designed to provide better support for open API clients like JDBC and ODBC.

3 min read
Back to Top ↑

jdbc

How-To : Connect HiveServer2 service with JDBC Client ?

HiveServer2 (HS2) is a server interface that enables remote clients to execute queries against Hive and retrieve the results. The current implementation, based on Thrift RPC, is an improved version of HiveServer and supports multi-client concurrency and authentication. It is designed to provide better support for open API clients like JDBC and ODBC.

3 min read
Back to Top ↑

cluster by

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read
Back to Top ↑

distribute by

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read
Back to Top ↑

hive interview

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read
Back to Top ↑

Hiveql

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read
Back to Top ↑

order by

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read
Back to Top ↑

sort by

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read
Back to Top ↑

spark sql

Hive : SORT BY vs ORDER BY vs DISTRIBUTE BY vs CLUSTER BY

In Apache Hive HQL, you can decide to order or sort your data differently based on ordering and distribution requirement. In this post we will look at how SORT BY, ORDER BY, DISTRIBUTE BY and CLUSTER BY behave differently in Hive. Let’s get started -

3 min read
Back to Top ↑

coprocessor

How-to : Write a CoProcessor in HBase

Coprocessor is a mechanism which helps to move computations closer to the data in HBase. It is like a Mapreduce framework to distribute tasks across the cluster.

2 min read
Back to Top ↑

storm

Back to Top ↑

twitter

Back to Top ↑

camus

Back to Top ↑

kafka-hdfs

Back to Top ↑

linkedin

Back to Top ↑

bg

Back to Top ↑

fg

Back to Top ↑

jobs

Back to Top ↑

process management

Back to Top ↑

unix

Back to Top ↑

collection

Hashmap Performance Improvements in Java 8

A developer-focused look at how Java 8 improved the performance of HashMap under high-collision scenarios, with code examples and practical explanations.

2 min read
Back to Top ↑

hashmap

Hashmap Performance Improvements in Java 8

A developer-focused look at how Java 8 improved the performance of HashMap under high-collision scenarios, with code examples and practical explanations.

2 min read
Back to Top ↑

fast

What is RDD in Spark ? and Why do we need it ?

A developer-friendly guide to understanding Resilient Distributed Datasets (RDDs) in Apache Spark, their properties, and why they are fundamental for fast, fault-tolerant distributed computing.

2 min read
Back to Top ↑

rdd

What is RDD in Spark ? and Why do we need it ?

A developer-friendly guide to understanding Resilient Distributed Datasets (RDDs) in Apache Spark, their properties, and why they are fundamental for fast, fault-tolerant distributed computing.

2 min read
Back to Top ↑

ELK Stack

My Book on ELK Stack : Learning ELK Stack

Learning ELK Stack

I am writing this post to announce the general availability of my book on ELK stack titled “ Learning ELK Stack “ with PacktPub publications. Book aims to provide individuals/technologists who seek to implement their own log and data analytics solutions using opensource stack of Elasticsearch, Logstash and Kibana popularly known as ELK stack.

1 min read
Back to Top ↑

Learning Elk Stack

My Book on ELK Stack : Learning ELK Stack

Learning ELK Stack

I am writing this post to announce the general availability of my book on ELK stack titled “ Learning ELK Stack “ with PacktPub publications. Book aims to provide individuals/technologists who seek to implement their own log and data analytics solutions using opensource stack of Elasticsearch, Logstash and Kibana popularly known as ELK stack.

1 min read
Back to Top ↑

PacktPub

My Book on ELK Stack : Learning ELK Stack

Learning ELK Stack

I am writing this post to announce the general availability of my book on ELK stack titled “ Learning ELK Stack “ with PacktPub publications. Book aims to provide individuals/technologists who seek to implement their own log and data analytics solutions using opensource stack of Elasticsearch, Logstash and Kibana popularly known as ELK stack.

1 min read
Back to Top ↑

Saurabh Chhajed

My Book on ELK Stack : Learning ELK Stack

Learning ELK Stack

I am writing this post to announce the general availability of my book on ELK stack titled “ Learning ELK Stack “ with PacktPub publications. Book aims to provide individuals/technologists who seek to implement their own log and data analytics solutions using opensource stack of Elasticsearch, Logstash and Kibana popularly known as ELK stack.

1 min read
Back to Top ↑

eclipse

Back to Top ↑

lambda

Back to Top ↑

maven

Back to Top ↑

mapereduce

Back to Top ↑

MultithreadedMapper

Back to Top ↑

spark interview

What does Skipped Stage means in Spark WebUI ?

Skipped Stages in Spark UI

You must have come across various scenarios where you see a DAG like below, where you see a few stages shows greyed out with a text (skipped) after the stage name. What does this mean? Did Spark ignore one of your stage due to an error? or this is due to something else? Well, it’s actually a good thing. It means that particular stage in the lineage DAG doesn’t need to be re-evaluated as its already evaluated and cached. This will save computation time for that stage. If you want to see what data frame for that stage was stored in the cache, you can check in Storage tab in Spark UI.

~1 min read
Back to Top ↑

spark on windows

Back to Top ↑

winutils.exe

Back to Top ↑

git

Back to Top ↑

repository

Back to Top ↑

split

Back to Top ↑

catboost

Back to Top ↑

HDFS

HDFS vs Object Storage: Deep Dive for Big Data Processing in the Cloud

With the rise of cloud-native data architectures and serverless computing, data engineers face a crucial decision when designing big data pipelines: HDFS or Object Storage? This decision impacts performance, scalability, fault tolerance, and cost. In this post, we’ll conduct a deep technical analysis of both storage systems and explore how they align with modern big data processing patterns on the cloud.

4 min read
Back to Top ↑

Object Storage

HDFS vs Object Storage: Deep Dive for Big Data Processing in the Cloud

With the rise of cloud-native data architectures and serverless computing, data engineers face a crucial decision when designing big data pipelines: HDFS or Object Storage? This decision impacts performance, scalability, fault tolerance, and cost. In this post, we’ll conduct a deep technical analysis of both storage systems and explore how they align with modern big data processing patterns on the cloud.

4 min read
Back to Top ↑

Spark

HDFS vs Object Storage: Deep Dive for Big Data Processing in the Cloud

With the rise of cloud-native data architectures and serverless computing, data engineers face a crucial decision when designing big data pipelines: HDFS or Object Storage? This decision impacts performance, scalability, fault tolerance, and cost. In this post, we’ll conduct a deep technical analysis of both storage systems and explore how they align with modern big data processing patterns on the cloud.

4 min read
Back to Top ↑

Big Data

HDFS vs Object Storage: Deep Dive for Big Data Processing in the Cloud

With the rise of cloud-native data architectures and serverless computing, data engineers face a crucial decision when designing big data pipelines: HDFS or Object Storage? This decision impacts performance, scalability, fault tolerance, and cost. In this post, we’ll conduct a deep technical analysis of both storage systems and explore how they align with modern big data processing patterns on the cloud.

4 min read
Back to Top ↑

Cloud

HDFS vs Object Storage: Deep Dive for Big Data Processing in the Cloud

With the rise of cloud-native data architectures and serverless computing, data engineers face a crucial decision when designing big data pipelines: HDFS or Object Storage? This decision impacts performance, scalability, fault tolerance, and cost. In this post, we’ll conduct a deep technical analysis of both storage systems and explore how they align with modern big data processing patterns on the cloud.

4 min read
Back to Top ↑

Data Engineering

HDFS vs Object Storage: Deep Dive for Big Data Processing in the Cloud

With the rise of cloud-native data architectures and serverless computing, data engineers face a crucial decision when designing big data pipelines: HDFS or Object Storage? This decision impacts performance, scalability, fault tolerance, and cost. In this post, we’ll conduct a deep technical analysis of both storage systems and explore how they align with modern big data processing patterns on the cloud.

4 min read
Back to Top ↑

data lakes

Unlocking Data Lakes with Apache Iceberg: The Open Table Format Revolutionizing Analytics

Why Apache Iceberg is the Game-Changer Your Data Lake Needs

Still relying on traditional data lakes with unreliable reads, lack of ACID guarantees, and painful schema evolution?

Welcome to Apache Iceberg—an open table format purpose-built for modern, petabyte-scale analytics. With native support for engines like Apache Spark, Trino, Presto, Hive, and Flink, Iceberg solves the biggest challenges in big data processing.

In this post, we’ll break down:

  • What Apache Iceberg is and why it matters
  • How its architecture works (with a visual!)
  • Key features that make it future-ready
  • Real-world use cases
  • A call to action for your next data platform decision
3 min read
Back to Top ↑

open table format

Unlocking Data Lakes with Apache Iceberg: The Open Table Format Revolutionizing Analytics

Why Apache Iceberg is the Game-Changer Your Data Lake Needs

Still relying on traditional data lakes with unreliable reads, lack of ACID guarantees, and painful schema evolution?

Welcome to Apache Iceberg—an open table format purpose-built for modern, petabyte-scale analytics. With native support for engines like Apache Spark, Trino, Presto, Hive, and Flink, Iceberg solves the biggest challenges in big data processing.

In this post, we’ll break down:

  • What Apache Iceberg is and why it matters
  • How its architecture works (with a visual!)
  • Key features that make it future-ready
  • Real-world use cases
  • A call to action for your next data platform decision
3 min read
Back to Top ↑

data architecture

Unlocking Data Lakes with Apache Iceberg: The Open Table Format Revolutionizing Analytics

Why Apache Iceberg is the Game-Changer Your Data Lake Needs

Still relying on traditional data lakes with unreliable reads, lack of ACID guarantees, and painful schema evolution?

Welcome to Apache Iceberg—an open table format purpose-built for modern, petabyte-scale analytics. With native support for engines like Apache Spark, Trino, Presto, Hive, and Flink, Iceberg solves the biggest challenges in big data processing.

In this post, we’ll break down:

  • What Apache Iceberg is and why it matters
  • How its architecture works (with a visual!)
  • Key features that make it future-ready
  • Real-world use cases
  • A call to action for your next data platform decision
3 min read
Back to Top ↑

Apache Kafka

Streaming from Kafka into an Iceberg Lakehouse, End to End

From Kafka Topics to a Lakehouse Table, the Right Way

Still landing Kafka data into HDFS or S3 as raw files and batching it later?

Back in 2015, I wrote about moving Twitter streams from Kafka to HDFS with Camus. A decade later, the goal is the same - get streaming data durably into storage - but the destination has changed completely. Today, we write straight into an Apache Iceberg table and get ACID commits, schema evolution, and time travel for free.

In this post, we’ll build the whole pipeline end to end:

  • A Kafka topic producing JSON events
  • A Spark Structured Streaming job that consumes the topic
  • Streaming writes into an Iceberg table with exactly-once semantics
  • Querying and maintaining the table (compaction, snapshots)
5 min read
Back to Top ↑

Spark Structured Streaming

Streaming from Kafka into an Iceberg Lakehouse, End to End

From Kafka Topics to a Lakehouse Table, the Right Way

Still landing Kafka data into HDFS or S3 as raw files and batching it later?

Back in 2015, I wrote about moving Twitter streams from Kafka to HDFS with Camus. A decade later, the goal is the same - get streaming data durably into storage - but the destination has changed completely. Today, we write straight into an Apache Iceberg table and get ACID commits, schema evolution, and time travel for free.

In this post, we’ll build the whole pipeline end to end:

  • A Kafka topic producing JSON events
  • A Spark Structured Streaming job that consumes the topic
  • Streaming writes into an Iceberg table with exactly-once semantics
  • Querying and maintaining the table (compaction, snapshots)
5 min read
Back to Top ↑

streaming

Streaming from Kafka into an Iceberg Lakehouse, End to End

From Kafka Topics to a Lakehouse Table, the Right Way

Still landing Kafka data into HDFS or S3 as raw files and batching it later?

Back in 2015, I wrote about moving Twitter streams from Kafka to HDFS with Camus. A decade later, the goal is the same - get streaming data durably into storage - but the destination has changed completely. Today, we write straight into an Apache Iceberg table and get ACID commits, schema evolution, and time travel for free.

In this post, we’ll build the whole pipeline end to end:

  • A Kafka topic producing JSON events
  • A Spark Structured Streaming job that consumes the topic
  • Streaming writes into an Iceberg table with exactly-once semantics
  • Querying and maintaining the table (compaction, snapshots)
5 min read
Back to Top ↑

Model Context Protocol

MCP for Data Engineers: Connecting AI Agents to Your Iceberg Lakehouse

Your AI Agent Can’t See Your Lakehouse - Let’s Fix That

How many times this year have you pasted a table schema into a chat window so the AI could help you write a query?

In my last post, we streamed Kafka events into an Apache Iceberg lakehouse with Spark Structured Streaming. The natural follow-up in 2026: instead of querying that table myself, can I just ask an AI agent what’s in it - and get a real answer from the live catalog, not a pasted screenshot?

That is exactly what the Model Context Protocol (MCP) is for. In this post, we’ll build a small MCP server that sits on top of an Iceberg catalog and exposes it to any MCP-capable agent (Claude, IDEs, your own agents):

  • What MCP actually is, in data-engineering terms
  • A working MCP server over an Iceberg catalog (Python, ~60 lines)
  • Wiring it into an agent host like Claude
  • The security lines you should not cross

Every code sample below was run end to end while writing this post - client, server, and a real Iceberg table.

8 min read
Back to Top ↑

MCP

MCP for Data Engineers: Connecting AI Agents to Your Iceberg Lakehouse

Your AI Agent Can’t See Your Lakehouse - Let’s Fix That

How many times this year have you pasted a table schema into a chat window so the AI could help you write a query?

In my last post, we streamed Kafka events into an Apache Iceberg lakehouse with Spark Structured Streaming. The natural follow-up in 2026: instead of querying that table myself, can I just ask an AI agent what’s in it - and get a real answer from the live catalog, not a pasted screenshot?

That is exactly what the Model Context Protocol (MCP) is for. In this post, we’ll build a small MCP server that sits on top of an Iceberg catalog and exposes it to any MCP-capable agent (Claude, IDEs, your own agents):

  • What MCP actually is, in data-engineering terms
  • A working MCP server over an Iceberg catalog (Python, ~60 lines)
  • Wiring it into an agent host like Claude
  • The security lines you should not cross

Every code sample below was run end to end while writing this post - client, server, and a real Iceberg table.

8 min read
Back to Top ↑

PyIceberg

MCP for Data Engineers: Connecting AI Agents to Your Iceberg Lakehouse

Your AI Agent Can’t See Your Lakehouse - Let’s Fix That

How many times this year have you pasted a table schema into a chat window so the AI could help you write a query?

In my last post, we streamed Kafka events into an Apache Iceberg lakehouse with Spark Structured Streaming. The natural follow-up in 2026: instead of querying that table myself, can I just ask an AI agent what’s in it - and get a real answer from the live catalog, not a pasted screenshot?

That is exactly what the Model Context Protocol (MCP) is for. In this post, we’ll build a small MCP server that sits on top of an Iceberg catalog and exposes it to any MCP-capable agent (Claude, IDEs, your own agents):

  • What MCP actually is, in data-engineering terms
  • A working MCP server over an Iceberg catalog (Python, ~60 lines)
  • Wiring it into an agent host like Claude
  • The security lines you should not cross

Every code sample below was run end to end while writing this post - client, server, and a real Iceberg table.

8 min read
Back to Top ↑

AI agents

MCP for Data Engineers: Connecting AI Agents to Your Iceberg Lakehouse

Your AI Agent Can’t See Your Lakehouse - Let’s Fix That

How many times this year have you pasted a table schema into a chat window so the AI could help you write a query?

In my last post, we streamed Kafka events into an Apache Iceberg lakehouse with Spark Structured Streaming. The natural follow-up in 2026: instead of querying that table myself, can I just ask an AI agent what’s in it - and get a real answer from the live catalog, not a pasted screenshot?

That is exactly what the Model Context Protocol (MCP) is for. In this post, we’ll build a small MCP server that sits on top of an Iceberg catalog and exposes it to any MCP-capable agent (Claude, IDEs, your own agents):

  • What MCP actually is, in data-engineering terms
  • A working MCP server over an Iceberg catalog (Python, ~60 lines)
  • Wiring it into an agent host like Claude
  • The security lines you should not cross

Every code sample below was run end to end while writing this post - client, server, and a real Iceberg table.

8 min read
Back to Top ↑

Claude

MCP for Data Engineers: Connecting AI Agents to Your Iceberg Lakehouse

Your AI Agent Can’t See Your Lakehouse - Let’s Fix That

How many times this year have you pasted a table schema into a chat window so the AI could help you write a query?

In my last post, we streamed Kafka events into an Apache Iceberg lakehouse with Spark Structured Streaming. The natural follow-up in 2026: instead of querying that table myself, can I just ask an AI agent what’s in it - and get a real answer from the live catalog, not a pasted screenshot?

That is exactly what the Model Context Protocol (MCP) is for. In this post, we’ll build a small MCP server that sits on top of an Iceberg catalog and exposes it to any MCP-capable agent (Claude, IDEs, your own agents):

  • What MCP actually is, in data-engineering terms
  • A working MCP server over an Iceberg catalog (Python, ~60 lines)
  • Wiring it into an agent host like Claude
  • The security lines you should not cross

Every code sample below was run end to end while writing this post - client, server, and a real Iceberg table.

8 min read
Back to Top ↑

Apache Spark

The Small-Files Problem: A Field Guide

The Maintenance Chapter Your Streaming Pipeline Was Missing

Your streaming job writes 2 MB Parquet files every two minutes. Six months later, your dashboard queries are ten times slower — and nobody changed the data. Sound familiar?

Yesterday, in Streaming from Kafka into an Iceberg Lakehouse, End to End, we built the whole pipeline — Kafka topic, Spark Structured Streaming, Iceberg table — and Step 6 ended with a warning: one-minute micro-batches create lots of small files — the classic streaming lakehouse problem. This post picks up exactly there. The pipeline works; the file layout is what’s rotting underneath it.

Same stack as yesterday: Kafka 4.3.1, Spark 4.2.0 (Scala 2.13), Iceberg 1.12.0, PySpark — and the same lakehouse.order_events table, partitioned by days(event_time).

In this post, we’ll:

  • See why small files hurt — with real numbers, not vibes

  • Diagnose the problem using Iceberg’s own metadata tables

  • Stop writing small files at write time

  • Fix the damage with compaction (rewrite_data_files, done right)

  • Keep the layout healthy with a checklist that fits on an index card

Step 1: Understand Why Small Files Hurt

It’s tempting to treat file count as cosmetic. It isn’t. Every file in your table is a tax paid four times: at plan time, at task-scheduling time, at request time, and at footer-read time.

Query planning. Iceberg query planning reads manifest files to decide which data files to scan — the manifest and snapshot layer I broke down in the Iceberg intro is exactly what gets stressed here. Dremio’s engineers measured this directly: the same dataset stored in 2,000 files of 256 MB each typically plans in under one second, while a table fragmented into 100,000 files pays a planning cost an order of magnitude higher. Manifests fragment too — every micro-batch commit adds manifests, and the metadata tree stops being an index and becomes a maze.

Task scheduling. Each file becomes at least one execution task. Scheduling costs roughly 1–10 milliseconds per task — trivial, until you multiply: a query against 100,000 files creates at least 100,000 tasks. Even if partition pruning eliminates 95% of them, the surviving 5,000 tasks carry 5–50 seconds of pure scheduling overhead before any data moves. On JVM engines like Spark, giant task graphs also pile on GC pressure, adding unpredictable latency.

Object-store requests. S3 charges $0.0004 per 1,000 GET requests. Reading a file takes at minimum one GET, but Parquet readers also fetch the footer and individual row groups — conservatively, two requests per file. So 100,000 files means 200,000 requests, or $0.08 per query, in storage API costs alone. A table queried 10,000 times a day burns $800 a day — $24,000 a month — on requests. Compact to 2,000 files and that drops to about $1,600 a month, a 93% reduction (source).

Footer reads. Every Parquet file carries a footer — schema, row-group metadata, column statistics. Reading it is mandatory and costs roughly the same whether the file is 1 MB or 512 MB: one to several milliseconds plus a network round trip. For a 256 MB file that’s noise; for a 1 MB file it dominates. If 95% of your read time goes to footers instead of column data, you have a severe small-files problem (source).

How fast it accumulates. Streaming engines checkpoint every 1–10 minutes, and each checkpoint commits new Parquet files — one per partition. A table with 100 partitions on 10-minute checkpoints produces 14,400 new files per day, most of them a few megabytes each (source). A table with 500,000 small files queried 50 times a day generates roughly 25 million GET requests a month. The write path that felt so elegant on day one is a liability by month six.

Step 2: Diagnose — Is This Actually Your Problem?

Don’t compact on vibes. Iceberg gives you metadata tables that answer the question directly:

-- File health at a glance: count, average size, smallest/largest
SELECT
  COUNT(*)                                        AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)     AS avg_mb,
  ROUND(MIN(file_size_in_bytes) / 1048576, 2)     AS min_mb,
  ROUND(MAX(file_size_in_bytes) / 1048576, 2)     AS max_mb,
  SUM(CASE WHEN file_size_in_bytes < 134217728
           THEN 1 ELSE 0 END)                     AS files_under_128mb
FROM lakehouse.order_events.files
WHERE content = 0;   -- 0 = DATA files; 1/2 = delete files

If a large share of files sits under 128 MB — or you see thousands of files averaging a few MB — you have the problem. Next, find where it hurts most, so your first compaction pass buys the biggest win:

-- Worst partitions first: file count and avg size per partition
SELECT
  partition,
  COUNT(*)                                     AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)  AS avg_mb
FROM lakehouse.order_events.files
WHERE content = 0
GROUP BY partition
ORDER BY file_count DESC
LIMIT 30;

Two more signals worth checking:

-- Are snapshots piling up from streaming appends?
SELECT snapshot_id, committed_at, operation, summary
FROM lakehouse.order_events.snapshots
ORDER BY committed_at DESC
LIMIT 20;

A wall of append snapshots with no replace (compaction) in sight confirms the diagnosis. And don’t forget the manifests metadata table — if manifest count is climbing with file count, planning overhead is compounding.

Outside SQL, the symptoms are just as telling: Spark UI jobs where a stage fans out into tens of thousands of tiny tasks, query plans that take longer than the scan itself, and — the one people notice last — your S3 GetObject/ListObjects request metrics climbing month over month while data volume stays flat.

Step 3: Fix It at Write Time

Compaction is corrective. The cheaper move is to stop writing small files in the first place. Three things worth knowing about the write path:

1. Target file size. The table property write.target-file-size-bytes defaults to 512 MB. Spark writers roll a new file when buffered data crosses this threshold — but only if a single task actually accumulates that much. A streaming micro-batch with 50 MB of data never reaches it, which is exactly how you get 2 MB files:

-- Set once at the table level (256 MB is a common streaming-table choice)
ALTER TABLE lakehouse.order_events
  SET TBLPROPERTIES ('write.target-file-size-bytes' = '268435456');

You can also override it per write with the target-file-size-bytes DataFrame option.

2. Distribution mode. write.distribution-mode decides how rows are shuffled before writing: none (no shuffle — each task writes whatever it has; the small-files factory), hash (hash-distribute by partition key), or range (range-distribute by partition or sort key). For streaming writes into a partitioned table, hash is usually the right call:

(df.writeStream
   .format("iceberg")
   .option("checkpointLocation", checkpoint_path)
   .option("distribution-mode", "hash")   # shuffle by partition key before writing
   .option("fanout-enabled", "true")      # keep partition writers open across batches
   .toTable("lakehouse.order_events"))

3. The fanout writer. Streaming data arrives out of order across partitions. Without fanout, every partition switch inside a task closes the current file and opens a new one — file rollover churn. write.spark.fanout.enabled (default false) keeps multiple partition writers open simultaneously, at the cost of more memory. For streaming jobs, enable it.

We used fanout-enabled: true in the streaming job from yesterday’s post for exactly this reason — without it, out-of-order events across days would have multiplied our file count from the very first batch.

Two more write-path habits that pay off:

  • repartition before batch writes. Line writers up with the partition spec so each task writes fewer, larger files:
from pyspark.sql.functions import to_date

def write_batch(batch_df, batch_id):
    (batch_df
       .repartition(to_date("event_time"))  # align writers with days(event_time)
       .writeTo("lakehouse.order_events")
       .option("fanout-enabled", "true")
       .append())

(query.writeStream
   .foreachBatch(write_batch)
   .option("checkpointLocation", checkpoint_path)
   .trigger(availableNow=True)
   .start())
  • Trigger intervals. The one-minute trigger we used yesterday is great for freshness demos and terrible for file layout — on a low-volume topic it guarantees tiny files. Prefer Trigger.AvailableNow or multi-minute processing-time triggers, and coalesce inside the batch function as above.

None of this eliminates compaction entirely — streaming tables will always drift — but it slows the drift from days to weeks.

Step 4: Compact What’s Already There

Yesterday’s post showed the one-line version — CALL lakehouse.system.rewrite_data_files(table => 'order_events'). Here’s the full picture. Iceberg’s compaction primitive rewrites selected data files into fewer, larger ones and commits the result as a new snapshot — fully online, no downtime; readers keep working against the old snapshot until the new one commits.

Binpack (default): fast and dumb, in a good way. It groups small files by partition and concatenates them up to the target size. No shuffle, no reordering — the cheapest possible rewrite:

-- Compact only the files that need it: small files in recent partitions
CALL lakehouse.system.rewrite_data_files(
  table    => 'order_events',
  strategy => 'binpack',
  where    => 'event_time >= current_timestamp() - INTERVAL 7 DAYS',
  options  => map(
    'target-file-size-bytes', '268435456',   -- 256 MB
    'min-input-files',        '5'             -- skip groups already healthy
  )
);

Sort: slower, but fixes layout too. Binpack doesn’t touch data ordering. If your queries filter on event_time and customer_id but files were written in arrival order, column statistics are weak and pruning suffers. The sort strategy globally sorts before allocating files to tasks, tightening min/max ranges per file:

-- Merge + cluster by the columns your queries actually filter on
CALL lakehouse.system.rewrite_data_files(
  table      => 'order_events',
  strategy   => 'sort',
  sort_order => 'event_time ASC NULLS LAST, customer_id ASC NULLS LAST',
  options    => map(
    'target-file-size-bytes',             '268435456',
    'partial-progress.enabled',           'true',
    'max-concurrent-file-group-rewrites', '5'
  )
);

-- For multi-column filters with no single dominant column, Z-order instead:
-- sort_order => 'zorder(event_time, customer_id)'

(Iceberg 1.12 also adds Hilbert-curve clustering as a newer alternative to Z-ordering — worth evaluating once your engine supports it.)

The options that matter in production:

Option Default What it does
target-file-size-bytes table’s write.target-file-size-bytes (512 MB) Output file size target
min-input-files 5 File groups with fewer files are skipped
min-file-size-bytes 75% of target Files below this are always candidates
max-file-size-bytes 180% of target Files above this are always candidates
max-concurrent-file-group-rewrites 5 Parallelism of the rewrite; raise for large tables
max-file-group-size-bytes 100 GB Splits the job into file groups
partial-progress.enabled false Commit file groups as they finish instead of one big commit
partial-progress.max-commits 10 Cap on commits when partial progress is on
rewrite-all false Force-rewrite everything, ignoring the filters above
rewrite-job-order none Order groups: bytes-asc, bytes-desc, files-asc, files-desc

Two of these deserve emphasis. partial-progress.enabled=true is the difference between losing hours of rewrite work on a mid-job failure and keeping what finished — always turn it on for large tables. And max-concurrent-file-group-rewrites is the knob real teams reach for at scale: Ancestry’s data team raised it from the default 5 to 100 to compact a 100-billion-row Iceberg table, because the default couldn’t keep up with their update frequency.

The follow-ups people forget. Compaction creates a new snapshot but leaves the old files on disk — old snapshots pin them. Run the cleanup chain, in this order:

-- 1. Expire old snapshots so their files can actually be deleted
--    (defaults: older_than = 5 days ago, retain_last = 1)
CALL lakehouse.system.expire_snapshots(
  table       => 'order_events',
  older_than  => TIMESTAMP '2026-10-01 00:00:00',
  retain_last => 10
);

-- 2. Consolidate manifest files (metadata-only, fast)
CALL lakehouse.system.rewrite_manifests(table => 'order_events');

-- 3. Sweep up orphan files from failed/partial writes (default window: 3 days)
--    Use dry_run first, and never set older_than shorter than your longest write.
CALL lakehouse.system.remove_orphan_files(
  table      => 'order_events',
  older_than => TIMESTAMP '2026-10-10 00:00:00',
  dry_run    => true
);

Compacting without ever expiring snapshots means storage grows forever. And remove_orphan_files with too aggressive a window can delete files from in-flight writes — multi-day windows are normal; hours are not.

Step 5: Keep It Healthy — the Index-Card Checklist

The industry rule of thumb is 128–512 MB per file for analytical workloads, and Iceberg’s own defaults agree: write.target-file-size-bytes is 512 MB, and the read path combines splits toward 128 MB (read.split.target-size).

Five lines, taped to your monitor:

  1. Measure first. Query tbl.files monthly (weekly for streaming tables). Alert when average file size drops below ~128 MB or file count crosses your threshold.
  2. Fix the writer. Set write.target-file-size-bytes, use distribution-mode=hash and fanout-enabled=true on streaming writes, and size your trigger intervals to the data volume.
  3. Compact on a cadence that matches ingestion. Streaming tables often need compaction every few hours, not nightly; batch tables can go weekly.
  4. Always expire snapshots and rewrite manifests after compacting. Compaction without cleanup is half a job.
  5. Sweep orphans quarterly with a safe older_than window — on mature streaming lakes, orphans routinely account for a quarter or more of billable storage (source).

Variations Worth Knowing

Delete-heavy tables (merge-on-read)?

Watch delete-file counts in tbl.files too — compact position deletes with rewrite_position_delete_files, not just data files.

Sub-hour freshness needs?

Don’t chase 512 MB if it means holding micro-batches open. 128–256 MB is fine for streaming tables where freshness matters more than scan efficiency.

Heavy point lookups?

Smaller files with tight clustering (sort/Z-order) beat larger unsorted ones. File size is a means to an end — the end is pruning.

✅ Conclusion

Small files are not a moral failing — they’re the natural output of a healthy streaming pipeline. The failure is letting the write path run unsupervised. Measure the file layout like you’d measure query latency, and the problem stays a maintenance task instead of becoming an incident.

If your Kafka-to-Iceberg pipeline is past the demo stage, its next chapter isn’t a new feature — it’s maintenance. Measure tbl.files, fix the writer, compact on cadence.

💬 Let’s Discuss

Are you compacting on a schedule, or only when queries get slow? Binpack or sort — which strategy do you reach for first? And what’s the worst small-files incident you’ve seen in production? Share your setup or war stories in the comments!

Additional Details

  • Description: A field guide to diagnosing, preventing, and fixing the small-files problem in Spark + Apache Iceberg lakehouse tables — with real numbers, working SQL, and a maintenance checklist.
11 min read
Back to Top ↑

small files

The Small-Files Problem: A Field Guide

The Maintenance Chapter Your Streaming Pipeline Was Missing

Your streaming job writes 2 MB Parquet files every two minutes. Six months later, your dashboard queries are ten times slower — and nobody changed the data. Sound familiar?

Yesterday, in Streaming from Kafka into an Iceberg Lakehouse, End to End, we built the whole pipeline — Kafka topic, Spark Structured Streaming, Iceberg table — and Step 6 ended with a warning: one-minute micro-batches create lots of small files — the classic streaming lakehouse problem. This post picks up exactly there. The pipeline works; the file layout is what’s rotting underneath it.

Same stack as yesterday: Kafka 4.3.1, Spark 4.2.0 (Scala 2.13), Iceberg 1.12.0, PySpark — and the same lakehouse.order_events table, partitioned by days(event_time).

In this post, we’ll:

  • See why small files hurt — with real numbers, not vibes

  • Diagnose the problem using Iceberg’s own metadata tables

  • Stop writing small files at write time

  • Fix the damage with compaction (rewrite_data_files, done right)

  • Keep the layout healthy with a checklist that fits on an index card

Step 1: Understand Why Small Files Hurt

It’s tempting to treat file count as cosmetic. It isn’t. Every file in your table is a tax paid four times: at plan time, at task-scheduling time, at request time, and at footer-read time.

Query planning. Iceberg query planning reads manifest files to decide which data files to scan — the manifest and snapshot layer I broke down in the Iceberg intro is exactly what gets stressed here. Dremio’s engineers measured this directly: the same dataset stored in 2,000 files of 256 MB each typically plans in under one second, while a table fragmented into 100,000 files pays a planning cost an order of magnitude higher. Manifests fragment too — every micro-batch commit adds manifests, and the metadata tree stops being an index and becomes a maze.

Task scheduling. Each file becomes at least one execution task. Scheduling costs roughly 1–10 milliseconds per task — trivial, until you multiply: a query against 100,000 files creates at least 100,000 tasks. Even if partition pruning eliminates 95% of them, the surviving 5,000 tasks carry 5–50 seconds of pure scheduling overhead before any data moves. On JVM engines like Spark, giant task graphs also pile on GC pressure, adding unpredictable latency.

Object-store requests. S3 charges $0.0004 per 1,000 GET requests. Reading a file takes at minimum one GET, but Parquet readers also fetch the footer and individual row groups — conservatively, two requests per file. So 100,000 files means 200,000 requests, or $0.08 per query, in storage API costs alone. A table queried 10,000 times a day burns $800 a day — $24,000 a month — on requests. Compact to 2,000 files and that drops to about $1,600 a month, a 93% reduction (source).

Footer reads. Every Parquet file carries a footer — schema, row-group metadata, column statistics. Reading it is mandatory and costs roughly the same whether the file is 1 MB or 512 MB: one to several milliseconds plus a network round trip. For a 256 MB file that’s noise; for a 1 MB file it dominates. If 95% of your read time goes to footers instead of column data, you have a severe small-files problem (source).

How fast it accumulates. Streaming engines checkpoint every 1–10 minutes, and each checkpoint commits new Parquet files — one per partition. A table with 100 partitions on 10-minute checkpoints produces 14,400 new files per day, most of them a few megabytes each (source). A table with 500,000 small files queried 50 times a day generates roughly 25 million GET requests a month. The write path that felt so elegant on day one is a liability by month six.

Step 2: Diagnose — Is This Actually Your Problem?

Don’t compact on vibes. Iceberg gives you metadata tables that answer the question directly:

-- File health at a glance: count, average size, smallest/largest
SELECT
  COUNT(*)                                        AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)     AS avg_mb,
  ROUND(MIN(file_size_in_bytes) / 1048576, 2)     AS min_mb,
  ROUND(MAX(file_size_in_bytes) / 1048576, 2)     AS max_mb,
  SUM(CASE WHEN file_size_in_bytes < 134217728
           THEN 1 ELSE 0 END)                     AS files_under_128mb
FROM lakehouse.order_events.files
WHERE content = 0;   -- 0 = DATA files; 1/2 = delete files

If a large share of files sits under 128 MB — or you see thousands of files averaging a few MB — you have the problem. Next, find where it hurts most, so your first compaction pass buys the biggest win:

-- Worst partitions first: file count and avg size per partition
SELECT
  partition,
  COUNT(*)                                     AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)  AS avg_mb
FROM lakehouse.order_events.files
WHERE content = 0
GROUP BY partition
ORDER BY file_count DESC
LIMIT 30;

Two more signals worth checking:

-- Are snapshots piling up from streaming appends?
SELECT snapshot_id, committed_at, operation, summary
FROM lakehouse.order_events.snapshots
ORDER BY committed_at DESC
LIMIT 20;

A wall of append snapshots with no replace (compaction) in sight confirms the diagnosis. And don’t forget the manifests metadata table — if manifest count is climbing with file count, planning overhead is compounding.

Outside SQL, the symptoms are just as telling: Spark UI jobs where a stage fans out into tens of thousands of tiny tasks, query plans that take longer than the scan itself, and — the one people notice last — your S3 GetObject/ListObjects request metrics climbing month over month while data volume stays flat.

Step 3: Fix It at Write Time

Compaction is corrective. The cheaper move is to stop writing small files in the first place. Three things worth knowing about the write path:

1. Target file size. The table property write.target-file-size-bytes defaults to 512 MB. Spark writers roll a new file when buffered data crosses this threshold — but only if a single task actually accumulates that much. A streaming micro-batch with 50 MB of data never reaches it, which is exactly how you get 2 MB files:

-- Set once at the table level (256 MB is a common streaming-table choice)
ALTER TABLE lakehouse.order_events
  SET TBLPROPERTIES ('write.target-file-size-bytes' = '268435456');

You can also override it per write with the target-file-size-bytes DataFrame option.

2. Distribution mode. write.distribution-mode decides how rows are shuffled before writing: none (no shuffle — each task writes whatever it has; the small-files factory), hash (hash-distribute by partition key), or range (range-distribute by partition or sort key). For streaming writes into a partitioned table, hash is usually the right call:

(df.writeStream
   .format("iceberg")
   .option("checkpointLocation", checkpoint_path)
   .option("distribution-mode", "hash")   # shuffle by partition key before writing
   .option("fanout-enabled", "true")      # keep partition writers open across batches
   .toTable("lakehouse.order_events"))

3. The fanout writer. Streaming data arrives out of order across partitions. Without fanout, every partition switch inside a task closes the current file and opens a new one — file rollover churn. write.spark.fanout.enabled (default false) keeps multiple partition writers open simultaneously, at the cost of more memory. For streaming jobs, enable it.

We used fanout-enabled: true in the streaming job from yesterday’s post for exactly this reason — without it, out-of-order events across days would have multiplied our file count from the very first batch.

Two more write-path habits that pay off:

  • repartition before batch writes. Line writers up with the partition spec so each task writes fewer, larger files:
from pyspark.sql.functions import to_date

def write_batch(batch_df, batch_id):
    (batch_df
       .repartition(to_date("event_time"))  # align writers with days(event_time)
       .writeTo("lakehouse.order_events")
       .option("fanout-enabled", "true")
       .append())

(query.writeStream
   .foreachBatch(write_batch)
   .option("checkpointLocation", checkpoint_path)
   .trigger(availableNow=True)
   .start())
  • Trigger intervals. The one-minute trigger we used yesterday is great for freshness demos and terrible for file layout — on a low-volume topic it guarantees tiny files. Prefer Trigger.AvailableNow or multi-minute processing-time triggers, and coalesce inside the batch function as above.

None of this eliminates compaction entirely — streaming tables will always drift — but it slows the drift from days to weeks.

Step 4: Compact What’s Already There

Yesterday’s post showed the one-line version — CALL lakehouse.system.rewrite_data_files(table => 'order_events'). Here’s the full picture. Iceberg’s compaction primitive rewrites selected data files into fewer, larger ones and commits the result as a new snapshot — fully online, no downtime; readers keep working against the old snapshot until the new one commits.

Binpack (default): fast and dumb, in a good way. It groups small files by partition and concatenates them up to the target size. No shuffle, no reordering — the cheapest possible rewrite:

-- Compact only the files that need it: small files in recent partitions
CALL lakehouse.system.rewrite_data_files(
  table    => 'order_events',
  strategy => 'binpack',
  where    => 'event_time >= current_timestamp() - INTERVAL 7 DAYS',
  options  => map(
    'target-file-size-bytes', '268435456',   -- 256 MB
    'min-input-files',        '5'             -- skip groups already healthy
  )
);

Sort: slower, but fixes layout too. Binpack doesn’t touch data ordering. If your queries filter on event_time and customer_id but files were written in arrival order, column statistics are weak and pruning suffers. The sort strategy globally sorts before allocating files to tasks, tightening min/max ranges per file:

-- Merge + cluster by the columns your queries actually filter on
CALL lakehouse.system.rewrite_data_files(
  table      => 'order_events',
  strategy   => 'sort',
  sort_order => 'event_time ASC NULLS LAST, customer_id ASC NULLS LAST',
  options    => map(
    'target-file-size-bytes',             '268435456',
    'partial-progress.enabled',           'true',
    'max-concurrent-file-group-rewrites', '5'
  )
);

-- For multi-column filters with no single dominant column, Z-order instead:
-- sort_order => 'zorder(event_time, customer_id)'

(Iceberg 1.12 also adds Hilbert-curve clustering as a newer alternative to Z-ordering — worth evaluating once your engine supports it.)

The options that matter in production:

Option Default What it does
target-file-size-bytes table’s write.target-file-size-bytes (512 MB) Output file size target
min-input-files 5 File groups with fewer files are skipped
min-file-size-bytes 75% of target Files below this are always candidates
max-file-size-bytes 180% of target Files above this are always candidates
max-concurrent-file-group-rewrites 5 Parallelism of the rewrite; raise for large tables
max-file-group-size-bytes 100 GB Splits the job into file groups
partial-progress.enabled false Commit file groups as they finish instead of one big commit
partial-progress.max-commits 10 Cap on commits when partial progress is on
rewrite-all false Force-rewrite everything, ignoring the filters above
rewrite-job-order none Order groups: bytes-asc, bytes-desc, files-asc, files-desc

Two of these deserve emphasis. partial-progress.enabled=true is the difference between losing hours of rewrite work on a mid-job failure and keeping what finished — always turn it on for large tables. And max-concurrent-file-group-rewrites is the knob real teams reach for at scale: Ancestry’s data team raised it from the default 5 to 100 to compact a 100-billion-row Iceberg table, because the default couldn’t keep up with their update frequency.

The follow-ups people forget. Compaction creates a new snapshot but leaves the old files on disk — old snapshots pin them. Run the cleanup chain, in this order:

-- 1. Expire old snapshots so their files can actually be deleted
--    (defaults: older_than = 5 days ago, retain_last = 1)
CALL lakehouse.system.expire_snapshots(
  table       => 'order_events',
  older_than  => TIMESTAMP '2026-10-01 00:00:00',
  retain_last => 10
);

-- 2. Consolidate manifest files (metadata-only, fast)
CALL lakehouse.system.rewrite_manifests(table => 'order_events');

-- 3. Sweep up orphan files from failed/partial writes (default window: 3 days)
--    Use dry_run first, and never set older_than shorter than your longest write.
CALL lakehouse.system.remove_orphan_files(
  table      => 'order_events',
  older_than => TIMESTAMP '2026-10-10 00:00:00',
  dry_run    => true
);

Compacting without ever expiring snapshots means storage grows forever. And remove_orphan_files with too aggressive a window can delete files from in-flight writes — multi-day windows are normal; hours are not.

Step 5: Keep It Healthy — the Index-Card Checklist

The industry rule of thumb is 128–512 MB per file for analytical workloads, and Iceberg’s own defaults agree: write.target-file-size-bytes is 512 MB, and the read path combines splits toward 128 MB (read.split.target-size).

Five lines, taped to your monitor:

  1. Measure first. Query tbl.files monthly (weekly for streaming tables). Alert when average file size drops below ~128 MB or file count crosses your threshold.
  2. Fix the writer. Set write.target-file-size-bytes, use distribution-mode=hash and fanout-enabled=true on streaming writes, and size your trigger intervals to the data volume.
  3. Compact on a cadence that matches ingestion. Streaming tables often need compaction every few hours, not nightly; batch tables can go weekly.
  4. Always expire snapshots and rewrite manifests after compacting. Compaction without cleanup is half a job.
  5. Sweep orphans quarterly with a safe older_than window — on mature streaming lakes, orphans routinely account for a quarter or more of billable storage (source).

Variations Worth Knowing

Delete-heavy tables (merge-on-read)?

Watch delete-file counts in tbl.files too — compact position deletes with rewrite_position_delete_files, not just data files.

Sub-hour freshness needs?

Don’t chase 512 MB if it means holding micro-batches open. 128–256 MB is fine for streaming tables where freshness matters more than scan efficiency.

Heavy point lookups?

Smaller files with tight clustering (sort/Z-order) beat larger unsorted ones. File size is a means to an end — the end is pruning.

✅ Conclusion

Small files are not a moral failing — they’re the natural output of a healthy streaming pipeline. The failure is letting the write path run unsupervised. Measure the file layout like you’d measure query latency, and the problem stays a maintenance task instead of becoming an incident.

If your Kafka-to-Iceberg pipeline is past the demo stage, its next chapter isn’t a new feature — it’s maintenance. Measure tbl.files, fix the writer, compact on cadence.

💬 Let’s Discuss

Are you compacting on a schedule, or only when queries get slow? Binpack or sort — which strategy do you reach for first? And what’s the worst small-files incident you’ve seen in production? Share your setup or war stories in the comments!

Additional Details

  • Description: A field guide to diagnosing, preventing, and fixing the small-files problem in Spark + Apache Iceberg lakehouse tables — with real numbers, working SQL, and a maintenance checklist.
11 min read
Back to Top ↑

compaction

The Small-Files Problem: A Field Guide

The Maintenance Chapter Your Streaming Pipeline Was Missing

Your streaming job writes 2 MB Parquet files every two minutes. Six months later, your dashboard queries are ten times slower — and nobody changed the data. Sound familiar?

Yesterday, in Streaming from Kafka into an Iceberg Lakehouse, End to End, we built the whole pipeline — Kafka topic, Spark Structured Streaming, Iceberg table — and Step 6 ended with a warning: one-minute micro-batches create lots of small files — the classic streaming lakehouse problem. This post picks up exactly there. The pipeline works; the file layout is what’s rotting underneath it.

Same stack as yesterday: Kafka 4.3.1, Spark 4.2.0 (Scala 2.13), Iceberg 1.12.0, PySpark — and the same lakehouse.order_events table, partitioned by days(event_time).

In this post, we’ll:

  • See why small files hurt — with real numbers, not vibes

  • Diagnose the problem using Iceberg’s own metadata tables

  • Stop writing small files at write time

  • Fix the damage with compaction (rewrite_data_files, done right)

  • Keep the layout healthy with a checklist that fits on an index card

Step 1: Understand Why Small Files Hurt

It’s tempting to treat file count as cosmetic. It isn’t. Every file in your table is a tax paid four times: at plan time, at task-scheduling time, at request time, and at footer-read time.

Query planning. Iceberg query planning reads manifest files to decide which data files to scan — the manifest and snapshot layer I broke down in the Iceberg intro is exactly what gets stressed here. Dremio’s engineers measured this directly: the same dataset stored in 2,000 files of 256 MB each typically plans in under one second, while a table fragmented into 100,000 files pays a planning cost an order of magnitude higher. Manifests fragment too — every micro-batch commit adds manifests, and the metadata tree stops being an index and becomes a maze.

Task scheduling. Each file becomes at least one execution task. Scheduling costs roughly 1–10 milliseconds per task — trivial, until you multiply: a query against 100,000 files creates at least 100,000 tasks. Even if partition pruning eliminates 95% of them, the surviving 5,000 tasks carry 5–50 seconds of pure scheduling overhead before any data moves. On JVM engines like Spark, giant task graphs also pile on GC pressure, adding unpredictable latency.

Object-store requests. S3 charges $0.0004 per 1,000 GET requests. Reading a file takes at minimum one GET, but Parquet readers also fetch the footer and individual row groups — conservatively, two requests per file. So 100,000 files means 200,000 requests, or $0.08 per query, in storage API costs alone. A table queried 10,000 times a day burns $800 a day — $24,000 a month — on requests. Compact to 2,000 files and that drops to about $1,600 a month, a 93% reduction (source).

Footer reads. Every Parquet file carries a footer — schema, row-group metadata, column statistics. Reading it is mandatory and costs roughly the same whether the file is 1 MB or 512 MB: one to several milliseconds plus a network round trip. For a 256 MB file that’s noise; for a 1 MB file it dominates. If 95% of your read time goes to footers instead of column data, you have a severe small-files problem (source).

How fast it accumulates. Streaming engines checkpoint every 1–10 minutes, and each checkpoint commits new Parquet files — one per partition. A table with 100 partitions on 10-minute checkpoints produces 14,400 new files per day, most of them a few megabytes each (source). A table with 500,000 small files queried 50 times a day generates roughly 25 million GET requests a month. The write path that felt so elegant on day one is a liability by month six.

Step 2: Diagnose — Is This Actually Your Problem?

Don’t compact on vibes. Iceberg gives you metadata tables that answer the question directly:

-- File health at a glance: count, average size, smallest/largest
SELECT
  COUNT(*)                                        AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)     AS avg_mb,
  ROUND(MIN(file_size_in_bytes) / 1048576, 2)     AS min_mb,
  ROUND(MAX(file_size_in_bytes) / 1048576, 2)     AS max_mb,
  SUM(CASE WHEN file_size_in_bytes < 134217728
           THEN 1 ELSE 0 END)                     AS files_under_128mb
FROM lakehouse.order_events.files
WHERE content = 0;   -- 0 = DATA files; 1/2 = delete files

If a large share of files sits under 128 MB — or you see thousands of files averaging a few MB — you have the problem. Next, find where it hurts most, so your first compaction pass buys the biggest win:

-- Worst partitions first: file count and avg size per partition
SELECT
  partition,
  COUNT(*)                                     AS file_count,
  ROUND(AVG(file_size_in_bytes) / 1048576, 2)  AS avg_mb
FROM lakehouse.order_events.files
WHERE content = 0
GROUP BY partition
ORDER BY file_count DESC
LIMIT 30;

Two more signals worth checking:

-- Are snapshots piling up from streaming appends?
SELECT snapshot_id, committed_at, operation, summary
FROM lakehouse.order_events.snapshots
ORDER BY committed_at DESC
LIMIT 20;

A wall of append snapshots with no replace (compaction) in sight confirms the diagnosis. And don’t forget the manifests metadata table — if manifest count is climbing with file count, planning overhead is compounding.

Outside SQL, the symptoms are just as telling: Spark UI jobs where a stage fans out into tens of thousands of tiny tasks, query plans that take longer than the scan itself, and — the one people notice last — your S3 GetObject/ListObjects request metrics climbing month over month while data volume stays flat.

Step 3: Fix It at Write Time

Compaction is corrective. The cheaper move is to stop writing small files in the first place. Three things worth knowing about the write path:

1. Target file size. The table property write.target-file-size-bytes defaults to 512 MB. Spark writers roll a new file when buffered data crosses this threshold — but only if a single task actually accumulates that much. A streaming micro-batch with 50 MB of data never reaches it, which is exactly how you get 2 MB files:

-- Set once at the table level (256 MB is a common streaming-table choice)
ALTER TABLE lakehouse.order_events
  SET TBLPROPERTIES ('write.target-file-size-bytes' = '268435456');

You can also override it per write with the target-file-size-bytes DataFrame option.

2. Distribution mode. write.distribution-mode decides how rows are shuffled before writing: none (no shuffle — each task writes whatever it has; the small-files factory), hash (hash-distribute by partition key), or range (range-distribute by partition or sort key). For streaming writes into a partitioned table, hash is usually the right call:

(df.writeStream
   .format("iceberg")
   .option("checkpointLocation", checkpoint_path)
   .option("distribution-mode", "hash")   # shuffle by partition key before writing
   .option("fanout-enabled", "true")      # keep partition writers open across batches
   .toTable("lakehouse.order_events"))

3. The fanout writer. Streaming data arrives out of order across partitions. Without fanout, every partition switch inside a task closes the current file and opens a new one — file rollover churn. write.spark.fanout.enabled (default false) keeps multiple partition writers open simultaneously, at the cost of more memory. For streaming jobs, enable it.

We used fanout-enabled: true in the streaming job from yesterday’s post for exactly this reason — without it, out-of-order events across days would have multiplied our file count from the very first batch.

Two more write-path habits that pay off:

  • repartition before batch writes. Line writers up with the partition spec so each task writes fewer, larger files:
from pyspark.sql.functions import to_date

def write_batch(batch_df, batch_id):
    (batch_df
       .repartition(to_date("event_time"))  # align writers with days(event_time)
       .writeTo("lakehouse.order_events")
       .option("fanout-enabled", "true")
       .append())

(query.writeStream
   .foreachBatch(write_batch)
   .option("checkpointLocation", checkpoint_path)
   .trigger(availableNow=True)
   .start())
  • Trigger intervals. The one-minute trigger we used yesterday is great for freshness demos and terrible for file layout — on a low-volume topic it guarantees tiny files. Prefer Trigger.AvailableNow or multi-minute processing-time triggers, and coalesce inside the batch function as above.

None of this eliminates compaction entirely — streaming tables will always drift — but it slows the drift from days to weeks.

Step 4: Compact What’s Already There

Yesterday’s post showed the one-line version — CALL lakehouse.system.rewrite_data_files(table => 'order_events'). Here’s the full picture. Iceberg’s compaction primitive rewrites selected data files into fewer, larger ones and commits the result as a new snapshot — fully online, no downtime; readers keep working against the old snapshot until the new one commits.

Binpack (default): fast and dumb, in a good way. It groups small files by partition and concatenates them up to the target size. No shuffle, no reordering — the cheapest possible rewrite:

-- Compact only the files that need it: small files in recent partitions
CALL lakehouse.system.rewrite_data_files(
  table    => 'order_events',
  strategy => 'binpack',
  where    => 'event_time >= current_timestamp() - INTERVAL 7 DAYS',
  options  => map(
    'target-file-size-bytes', '268435456',   -- 256 MB
    'min-input-files',        '5'             -- skip groups already healthy
  )
);

Sort: slower, but fixes layout too. Binpack doesn’t touch data ordering. If your queries filter on event_time and customer_id but files were written in arrival order, column statistics are weak and pruning suffers. The sort strategy globally sorts before allocating files to tasks, tightening min/max ranges per file:

-- Merge + cluster by the columns your queries actually filter on
CALL lakehouse.system.rewrite_data_files(
  table      => 'order_events',
  strategy   => 'sort',
  sort_order => 'event_time ASC NULLS LAST, customer_id ASC NULLS LAST',
  options    => map(
    'target-file-size-bytes',             '268435456',
    'partial-progress.enabled',           'true',
    'max-concurrent-file-group-rewrites', '5'
  )
);

-- For multi-column filters with no single dominant column, Z-order instead:
-- sort_order => 'zorder(event_time, customer_id)'

(Iceberg 1.12 also adds Hilbert-curve clustering as a newer alternative to Z-ordering — worth evaluating once your engine supports it.)

The options that matter in production:

Option Default What it does
target-file-size-bytes table’s write.target-file-size-bytes (512 MB) Output file size target
min-input-files 5 File groups with fewer files are skipped
min-file-size-bytes 75% of target Files below this are always candidates
max-file-size-bytes 180% of target Files above this are always candidates
max-concurrent-file-group-rewrites 5 Parallelism of the rewrite; raise for large tables
max-file-group-size-bytes 100 GB Splits the job into file groups
partial-progress.enabled false Commit file groups as they finish instead of one big commit
partial-progress.max-commits 10 Cap on commits when partial progress is on
rewrite-all false Force-rewrite everything, ignoring the filters above
rewrite-job-order none Order groups: bytes-asc, bytes-desc, files-asc, files-desc

Two of these deserve emphasis. partial-progress.enabled=true is the difference between losing hours of rewrite work on a mid-job failure and keeping what finished — always turn it on for large tables. And max-concurrent-file-group-rewrites is the knob real teams reach for at scale: Ancestry’s data team raised it from the default 5 to 100 to compact a 100-billion-row Iceberg table, because the default couldn’t keep up with their update frequency.

The follow-ups people forget. Compaction creates a new snapshot but leaves the old files on disk — old snapshots pin them. Run the cleanup chain, in this order:

-- 1. Expire old snapshots so their files can actually be deleted
--    (defaults: older_than = 5 days ago, retain_last = 1)
CALL lakehouse.system.expire_snapshots(
  table       => 'order_events',
  older_than  => TIMESTAMP '2026-10-01 00:00:00',
  retain_last => 10
);

-- 2. Consolidate manifest files (metadata-only, fast)
CALL lakehouse.system.rewrite_manifests(table => 'order_events');

-- 3. Sweep up orphan files from failed/partial writes (default window: 3 days)
--    Use dry_run first, and never set older_than shorter than your longest write.
CALL lakehouse.system.remove_orphan_files(
  table      => 'order_events',
  older_than => TIMESTAMP '2026-10-10 00:00:00',
  dry_run    => true
);

Compacting without ever expiring snapshots means storage grows forever. And remove_orphan_files with too aggressive a window can delete files from in-flight writes — multi-day windows are normal; hours are not.

Step 5: Keep It Healthy — the Index-Card Checklist

The industry rule of thumb is 128–512 MB per file for analytical workloads, and Iceberg’s own defaults agree: write.target-file-size-bytes is 512 MB, and the read path combines splits toward 128 MB (read.split.target-size).

Five lines, taped to your monitor:

  1. Measure first. Query tbl.files monthly (weekly for streaming tables). Alert when average file size drops below ~128 MB or file count crosses your threshold.
  2. Fix the writer. Set write.target-file-size-bytes, use distribution-mode=hash and fanout-enabled=true on streaming writes, and size your trigger intervals to the data volume.
  3. Compact on a cadence that matches ingestion. Streaming tables often need compaction every few hours, not nightly; batch tables can go weekly.
  4. Always expire snapshots and rewrite manifests after compacting. Compaction without cleanup is half a job.
  5. Sweep orphans quarterly with a safe older_than window — on mature streaming lakes, orphans routinely account for a quarter or more of billable storage (source).

Variations Worth Knowing

Delete-heavy tables (merge-on-read)?

Watch delete-file counts in tbl.files too — compact position deletes with rewrite_position_delete_files, not just data files.

Sub-hour freshness needs?

Don’t chase 512 MB if it means holding micro-batches open. 128–256 MB is fine for streaming tables where freshness matters more than scan efficiency.

Heavy point lookups?

Smaller files with tight clustering (sort/Z-order) beat larger unsorted ones. File size is a means to an end — the end is pruning.

✅ Conclusion

Small files are not a moral failing — they’re the natural output of a healthy streaming pipeline. The failure is letting the write path run unsupervised. Measure the file layout like you’d measure query latency, and the problem stays a maintenance task instead of becoming an incident.

If your Kafka-to-Iceberg pipeline is past the demo stage, its next chapter isn’t a new feature — it’s maintenance. Measure tbl.files, fix the writer, compact on cadence.

💬 Let’s Discuss

Are you compacting on a schedule, or only when queries get slow? Binpack or sort — which strategy do you reach for first? And what’s the worst small-files incident you’ve seen in production? Share your setup or war stories in the comments!

Additional Details

  • Description: A field guide to diagnosing, preventing, and fixing the small-files problem in Spark + Apache Iceberg lakehouse tables — with real numbers, working SQL, and a maintenance checklist.
11 min read
Back to Top ↑