Each product combination added another pipeline to maintain
Upsell and cross-sell propensity models are a core growth engine for a modern insurance and financial services provider. For one leading US life insurance enterprise, more than 20 distinct product upsell and cross-sell combinations were each served by a dedicated end-to-end machine learning pipeline, with model input tables reaching tens of millions of rows and 250+ features. These pipelines ran on a coupled storage and compute architecture on AWS, spread across multiple services and bound to single-node computation. Every additional product combination meant another pipeline to build, schedule, and maintain.
As part of a broader data modernization initiative, the organization decided to migrate its core data warehouse layer from Amazon Redshift to the Databricks Lakehouse. Rather than moving the warehouse alone, the engineering team transitioned both the core data layers and the downstream machine learning pipelines entirely into Databricks, consolidating compute, feature engineering, orchestration, model training, and storage under a single governance plane.
The result is a platform where the data that feeds the models, the code that trains them, and the jobs that run them share the same environment, the same access controls, and the same operational tooling.
Extending the migration to include downstream ML
The legacy environment had grown one pipeline at a time. Each product combination carried its own end-to-end pipeline, and each pipeline depended on several cloud services stitched together across network boundaries. The result was a set of compounding constraints: a multi-service architecture that added latency to every handoff, single-node computation that capped how far model input tables could grow, significant engineering overhead to keep the pieces running, complex networking between services, and governance fractured across systems.
These constraints reinforced one another. Because data lived in the warehouse while training ran elsewhere, every pipeline run depended on moving data across service boundaries. Because processing was bound to a single node, scaling meant provisioning a larger machine rather than adding workers, an approach with a hard limit as input tables reached tens of millions of rows and hundreds of features. And because access rules were defined separately in each service, there was no single place to answer who could see which data or which tables fed a given model.
The warehouse migration surfaced a strategic question. A decoupled hybrid approach would move the warehouse to Databricks while leaving the machine learning pipelines on the existing services, preserving the multi-service latency and administrative burden. A unified platform approach would bring everything into one ecosystem. The team chose the unified path based on four decision points: consolidating compute, feature engineering, orchestration, model training, and data storage; eliminating multi-service network latency; reducing administrative overhead; and establishing a robust governance plane. The hybrid option required less migration effort up front, but it would have left the organization operating two ecosystems indefinitely. The unified option meant refactoring the pipelines themselves, in exchange for removing the root causes of the friction rather than relocating them.
Migrating the data layer and ML pipelines in four stages
Moving more than 20 production machine learning pipelines alongside the warehouse layer required a phased approach that preserved model reproducibility at every step.
1. Data assets and compute requirements
The migration began with a full inventory of data assets and compute requirements. Legacy named-version tables were transitioned into Delta tables, using Delta Lake time travel to guarantee reproducible model training without maintaining manual table copies. Shared features were centralized in the Databricks Feature Store, managed volumes were set up for secure model artifact storage, and the compute environment was configured around production-ready job clusters. Defining these foundations first gave the refactored pipelines a stable target: versioned data, shared features, governed artifact storage, and standardized compute.
2. Codebase refactoring
The entire machine learning pipeline codebase was refactored from Polars to PySpark. This removed the single-node ceiling, allowing the pipelines to scale natively across multi-node Spark clusters as input tables and feature counts grow. Logic that previously ran in memory on one machine now executes as distributed jobs, reading and writing Delta tables directly on the platform where the data lives.
3. Workflow orchestration and parameterization
Execution and orchestration were modernized with Databricks Workflows. Jobs were configured with automated schedules, dependency mapping, and parameter injection. This dynamic configuration enabled a single robust codebase to be reused across all models, replacing per-product pipeline maintenance. Instead of maintaining a separate pipeline for each of the 20+ product combinations, the team parameterizes a single pipeline per run, so a fix or improvement made once applies to every model.
4. Model and data quality monitoring
The team established a monitoring framework using Databricks SQL Dashboards for model performance and data quality reporting, with Databricks SQL Alerts delivering notifications when metrics drift outside expected ranges. Because monitoring runs on the same platform as the data and the models, performance and data quality reporting draws on the same governed tables the pipelines produce, with no separate monitoring stack to maintain.
Changes to pipeline execution and maintenance
The transition to a unified Databricks Lakehouse delivered clear operational outcomes:



