Six hours of manual migration work in twenty minutes

A Pydantic AI multi-agent system that migrates any SQL source to Databricks: schema analysis, SQL to Unity Catalog conversion, documentation, S3 staged transfer, and automated validation. Benchmarked on seven tables and roughly ten million rows at 20 minutes against 6 hours by hand.

Problem

Moving a SQL source onto Databricks was six hours of manual work per pass. Someone read the schema, rewrote the SQL for Unity Catalog, staged the data, then checked the result by eye. The manual process left behind no lineage and no documentation, so the next person started from nothing.

Approach

I architected a Pydantic AI multi-agent system that runs inside Databricks notebooks and covers the whole path: schema analysis, SQL to Unity Catalog conversion, documentation generation, S3 staged data transfer, and automated validation. Each stage is an agent with a narrow job. Lineage and documentation are produced by the run rather than written up afterwards.

Architecture

A SQL source passes through schema analysis, Unity Catalog conversion with documentation generation, S3 staged transfer, and automated validation before landing in Databricks with full lineage.

The hard part

A migration nobody can check is not a migration, so the validation agent had to be as serious as the conversion agent. The other constraint was repeatability: the system had to work against any SQL source, not the one in front of us, because a one-off script would have been faster to write and worthless the second time. That constraint is what let the team scale from 2 to 7 developers without every migration becoming bespoke.

Outcome

Benchmarked on seven representative tables and roughly ten million rows combined, the run took twenty minutes end to end against six hours by hand, and produced lineage and documentation the manual process never did. It became a repeatable migration framework reused across the engagement. The result also triggered a follow-on Databricks to GCP migration sale.

Next

Cutting an air cargo forecast cycle from six weeks to minutes