Unified ML on Databricks

A U.S. life insurer unified 20+ ML pipelines on Databricks, using distributed Spark and reusable code to simplify maintenance.

Key Takeaways:

20+ ML Pipelines Consolidated on Databricks
Distributed PySpark Replaces Single-Node Processing
One Parameterized Codebase Simplifies Model Maintenance

Each product combination added another pipeline to maintain

Upsell and cross-sell propensity models are a core growth engine for a modern insurance and financial services provider. For one leading US life insurance enterprise, more than 20 distinct product upsell and cross-sell combinations were each served by a dedicated end-to-end machine learning pipeline, with model input tables reaching tens of millions of rows and 250+ features. These pipelines ran on a coupled storage and compute architecture on AWS, spread across multiple services and bound to single-node computation. Every additional product combination meant another pipeline to build, schedule, and maintain.

As part of a broader data modernization initiative, the organization decided to migrate its core data warehouse layer from Amazon Redshift to the Databricks Lakehouse. Rather than moving the warehouse alone, the engineering team transitioned both the core data layers and the downstream machine learning pipelines entirely into Databricks, consolidating compute, feature engineering, orchestration, model training, and storage under a single governance plane.

The result is a platform where the data that feeds the models, the code that trains them, and the jobs that run them share the same environment, the same access controls, and the same operational tooling.

‍

Extending the migration to include downstream ML

The legacy environment had grown one pipeline at a time. Each product combination carried its own end-to-end pipeline, and each pipeline depended on several cloud services stitched together across network boundaries. The result was a set of compounding constraints: a multi-service architecture that added latency to every handoff, single-node computation that capped how far model input tables could grow, significant engineering overhead to keep the pieces running, complex networking between services, and governance fractured across systems.
‍

These constraints reinforced one another. Because data lived in the warehouse while training ran elsewhere, every pipeline run depended on moving data across service boundaries. Because processing was bound to a single node, scaling meant provisioning a larger machine rather than adding workers, an approach with a hard limit as input tables reached tens of millions of rows and hundreds of features. And because access rules were defined separately in each service, there was no single place to answer who could see which data or which tables fed a given model.
‍

The warehouse migration surfaced a strategic question. A decoupled hybrid approach would move the warehouse to Databricks while leaving the machine learning pipelines on the existing services, preserving the multi-service latency and administrative burden. A unified platform approach would bring everything into one ecosystem. The team chose the unified path based on four decision points: consolidating compute, feature engineering, orchestration, model training, and data storage; eliminating multi-service network latency; reducing administrative overhead; and establishing a robust governance plane. The hybrid option required less migration effort up front, but it would have left the organization operating two ecosystems indefinitely. The unified option meant refactoring the pipelines themselves, in exchange for removing the root causes of the friction rather than relocating them.

‍

Migrating the data layer and ML pipelines in four stages

Moving more than 20 production machine learning pipelines alongside the warehouse layer required a phased approach that preserved model reproducibility at every step.

‍

1. Data assets and compute requirements

The migration began with a full inventory of data assets and compute requirements. Legacy named-version tables were transitioned into Delta tables, using Delta Lake time travel to guarantee reproducible model training without maintaining manual table copies. Shared features were centralized in the Databricks Feature Store, managed volumes were set up for secure model artifact storage, and the compute environment was configured around production-ready job clusters. Defining these foundations first gave the refactored pipelines a stable target: versioned data, shared features, governed artifact storage, and standardized compute.

‍

2. Codebase refactoring

The entire machine learning pipeline codebase was refactored from Polars to PySpark. This removed the single-node ceiling, allowing the pipelines to scale natively across multi-node Spark clusters as input tables and feature counts grow. Logic that previously ran in memory on one machine now executes as distributed jobs, reading and writing Delta tables directly on the platform where the data lives.

‍

3. Workflow orchestration and parameterization

Execution and orchestration were modernized with Databricks Workflows. Jobs were configured with automated schedules, dependency mapping, and parameter injection. This dynamic configuration enabled a single robust codebase to be reused across all models, replacing per-product pipeline maintenance. Instead of maintaining a separate pipeline for each of the 20+ product combinations, the team parameterizes a single pipeline per run, so a fix or improvement made once applies to every model.

‍

4. Model and data quality monitoring

The team established a monitoring framework using Databricks SQL Dashboards for model performance and data quality reporting, with Databricks SQL Alerts delivering notifications when metrics drift outside expected ranges. Because monitoring runs on the same platform as the data and the models, performance and data quality reporting draws on the same governed tables the pipelines produce, with no separate monitoring stack to maintain.

‍

Changes to pipeline execution and maintenance

The transition to a unified Databricks Lakehouse delivered clear operational outcomes:

Unified Architecture

Data storage, feature engineering, model training, and orchestration now live in one ecosystem, eliminating multi-service network latency and cross-service handoffs. The warehouse layer and the models it feeds are no longer separated by service boundaries.

Scalable, Distributed Processing

Pipelines that were bound to a single node now scale across multi-node Spark clusters, removing the ceiling on model input size. Capacity now grows by adding workers to a cluster rather than by resizing a single machine.

Reduced Administrative Overhead

Simplified DAG orchestration and a robust, reusable codebase replaced the maintenance of 20+ individually managed pipelines. Schedules, dependencies, and parameters are managed in one orchestration layer.

A Single Data Governance Plane

Tables, features, and model artifacts are governed in one place, replacing fractured controls spread across services. Delta Lake time travel adds reproducibility, so a training run can be traced back to the exact version of the data it used.

By treating the warehouse migration as an opportunity to unify rather than simply relocate, the organization turned a collection of independently maintained pipelines into a single, scalable machine learning platform. With infrastructure friction removed, the data science team can direct its effort toward improving models and expanding to new product combinations.

The engagement also shows that a warehouse migration is rarely just a warehouse decision. Moving the core data layer while leaving the machine learning pipelines behind would have preserved the latency, the administrative burden, and the fractured governance that motivated the modernization in the first place. Bringing both into the same ecosystem meant the refactoring effort was spent once, and its benefits now apply to every pipeline that runs on the platform.
‍

The foundations put in place during the migration are built to carry future growth. Versioned Delta tables, a centralized feature store, parameterized workflows, and built-in monitoring give each new model the same reproducibility, scalability, and governance as the existing ones, without adding another pipeline to maintain. What began as a Redshift migration became the base for a machine learning practice that scales with the business rather than with engineering headcount.

Want to discuss a solution for you?
Learn More
Elite engineers ready to accelerate your roadmap
Start vetting within one week
Have talent placed in under a month.

Continue Reading

Databricks geospatial data platform for telecom network planning and infrastructure analysis

Unified Geospatial Intelligence

Network Planning From Weeks to Minutes

Developer infrastructure modernization illustrating the transition from cloud virtual machines to standardized local development environments.

Developer Infrastructure Savings

70% Lower Costs. 80% Fewer VMs