When Network Planning Takes Weeks
Network planning is the process of evaluating, designing, and prioritizing telecom infrastructure deployments such as fiber expansion, copper-to-fiber upgrades, and wireless enhancements. A major consultancy advising leading telecommunications carriers has to cover a lot of ground in every engagement: network coverage and demand, infrastructure gaps, improvement feasibility and cost, engineering and regulatory constraints, and recommended deployment areas ranked by ROI and impact. Network design recommendations are an additional differentiator.
Traditionally this was a multi-week workflow. Experts performed on-site surveys to assess physical space, power availability, safety, and accessibility, then manually analyzed signal strength, throughput measurements, infrastructure maps, and site photos. Behind that work sat a siloed data architecture, with geospatial data fragmented across Amazon Redshift, PostgreSQL, object storage, and on-premise GIS software.
To cut turnaround time and logistics costs, and to put the data in the hands of the people who need it, the engineering team moved the entire stack to a Databricks Lakehouse governed by Unity Catalog, validated each layer against the legacy environment, and switched Redshift off.
Four Systems, Four Sets of Rules
Day to day, the old stack ran on multi-layer orchestrated systems and several independently owned processes. Initial efforts to reduce silos through federated queries, external schemas, and archiving standardized how the systems talked to each other, but key limitations remained.
Access controls and permission boundaries had to be established and maintained separately on each data system, and integrating assets across systems introduced a new network security challenge each time. Geospatial production pipelines were slow because of limited parallelism in the database and desktop GIS engines, and a limited number of GIS licenses restricted who could use cloud computing resources. Experimentation happened on local machines, isolated from data split across platforms, and large file transfers had to pass through individual PCs.
Each problem fed the next. With data split across platforms, no single engine could see everything a geospatial analysis needed, so data had to be moved before it could be combined. Moving it required new network paths and permissions to secure, and maintaining those rules on every system multiplied the administrative work. Good platform administration kept security under control, but geospatial performance at scale remained a bottleneck that administration alone could not remove.
Business users were affected too. Analysts could not query the data directly and depended on curated reports built by technical developers. An internal tool had been designed over roughly three months to move tables between Redshift, Redshift Spectrum, and PostgreSQL, but non-technical analysts could not access the full data without mastering it.
Why Databricks
After benchmarking data platform alternatives, the team selected Databricks for three reasons. The first was governance: Unity Catalog covers every asset type in one place and adds lineage, usage, and impact visibility. The second was performance: data and compute live on the same platform, with Spark, Photon, MLflow, native spatial (ST) functions, grid indexing, managed orchestration, and serverless resources, while Delta Lake brings time travel, geospatial data support, intelligent data skipping, auto-compaction, and Auto Loader. The third was accessibility: a gentle learning curve and built-in data intelligence put the platform within reach of non-technical users. Each reason maps to something that had been holding the consultancy back: governance fragmented across systems, compute separated from data, and a platform that only technical specialists could use.
How the Migration Unfolded
The core migration, from platform adoption through the Redshift switch-off, was completed in roughly six months, with continuous validation between the legacy and target environments.
Phase 1: Namespaces and Access Controls
In about one week, the team migrated the legacy database and schema structure into Unity Catalog catalogs and schemas. The mapping was not always one-to-one at the catalog level, so the structure was redesigned where needed. User groups were created and permissions assigned accordingly. Starting here ensured that every table arriving later was already covered by one permission model instead of a separate boundary per system.
Phase 2: Full Backfill at the Raw Layer
Over approximately one month, all Redshift tables were migrated in a single batch through Redshift Spectrum, and the upstream incremental and batch ingestions were replicated. Scheduled monitoring queries compared Redshift and Unity Catalog data. More than 100 tables across around 20 processes were replicated successfully at the bronze level along with their upstream processes. Running both environments side by side with scheduled comparisons gave the team objective evidence of parity before any consumer was moved.
Phase 3: Transformation Pipelines
In roughly one more month, existing dbt transformation processes were rebuilt as Delta Live Tables pipelines and geospatial ST scripts. Monitoring queries again validated the Redshift curated layer against the new silver layer in Unity Catalog. With distributed compute now behind the transformations, the parallelism limits that had slowed the geospatial pipelines no longer applied.
Phase 4: Users and BI Tools
Over about three months, technical users were moved to work directly on Unity Catalog and BI tools were repointed to Databricks SQL Warehouses. This was the longest phase of the core migration, reflecting the work of moving people and tools rather than tables.
Phase 5: Switching Off Redshift
In approximately two weeks, upstream and downstream processes in Redshift were retired, a final snapshot was taken, and the cluster was paused. By this point the data had been validated, the pipelines rebuilt, and every consumer was working on Databricks, so nothing depended on the old warehouse any longer.
Phase 6: Opening the Data to Non-Technical Users
In a follow-on phase of about three months, dashboards were replicated in Databricks, metric views were created, and Genie Spaces were enabled so business users could explore data in natural language. This closed the gap the earlier internal tool could not: analysts no longer needed to master a technical tool or wait for a developer-built report to reach the data.
What Changed for the Business and the Platform
The platform also made a new kind of deliverable possible. On top of the governed data, the team built a data product for instant network planning: a user draws a boundary on a map and sets preferences, the application triggers a Databricks job, and a pipeline combines a curated geospatial dataset with trained models to return a report on coverage, demand, restrictions, feasibility, ROI, and design. The curated dataset brings together vector and satellite raster layers under H3 indexing, along with a vision model trained on street-level imagery that field experts tagged for features such as utility poles and antennas. Much of what a site visit used to capture is now in the data.
The effects showed up on both the business and the technical side:



