Reliable analytics depend on more than having data available. Enterprise teams also need to understand how that data enters the platform, where failures occur, and how ingestion changes affect downstream systems.
For this organization, a third-party provider managed a critical part of the ingestion process. That arrangement limited the internal team’s visibility into how data was processed, monitored, and validated before reaching analytics users.
Factored is working with the company to bring that capability into its own Azure environment. The engagement is rebuilding the ingestion layer on Azure Databricks so the internal team can operate, monitor, and extend the infrastructure directly rather than depending on an external provider.
The migration is still in progress. Its target is both technical and financial: establish an internally managed ingestion platform and enable the company to retire an external contract reported at approximately hundreds of thousands per year.
The vendor-managed ingestion layer limited operational visibility
The existing architecture created a gap between source systems and the teams consuming their data.
Because ingestion was managed outside the company, internal engineers had limited control over pipeline behavior, monitoring, and validation. That made it harder to investigate data-quality issues and understand the condition of ingestion pipelines without relying on the provider.
The arrangement also carried a significant recurring cost. The company reports spending hundreds of thousands annually on the external ingestion contract.
A third constraint was architectural. Multiple sources of truth complicated efforts to create a consistent view of business and operational data. Moving ingestion in-house therefore required more than transferring existing pipelines. The replacement needed to give the company greater control over how data enters its environment and how those processes are operated over time.
The current target is to complete the migration by the end of the quarter and avoid renewing the third-party ingestion contract.
The new architecture puts ingestion inside the company’s Azure environment
Factored is rebuilding the ingestion layer with Azure Databricks as a core processing and governance environment.
The design uses declarative streaming pipelines and incremental ingestion patterns intended to support ongoing data updates without reproducing the constraints of the existing vendor-managed process. Databricks Lakeflow and custom connectors are being used for ingestion, while materialized views support downstream data consumption and query performance.
Azure Data Factory provides orchestration across the environment.
Databricks Asset Bundles are also part of the implementation. They allow workflow and environment configuration to be managed through version-controlled definitions rather than relying solely on manually configured resources. For the internal team, that is important to the broader ownership objective: the company needs to be able to reproduce, modify, and operate the ingestion environment after the migration is complete.
The implementation currently includes:
- Orchestration: Azure Data Factory
- Compute and governance: Azure Databricks
- Deployment configuration: Databricks Asset Bundles
- Ingestion: Databricks Lakeflow, declarative streaming pipelines, and custom connectors
- Downstream performance: Materialized views
Migration speed has to be balanced with long-term operability
Replacing an external ingestion service introduces a different engineering problem from building an isolated new pipeline.
The team has to move existing ingestion responsibilities without creating new reliability problems for downstream consumers. At the same time, reproducing the vendor’s implementation exactly would undermine part of the reason for internalizing the platform in the first place.
The central trade-off is therefore between migration speed and maintainability.
Factored’s implementation is designed around infrastructure that the company’s own team can inspect, version, monitor, and extend. That requires treating operational ownership as an architectural requirement rather than as a handoff activity after the migration.
It also changes what “done” means. Completing pipeline migration is only one part of the objective. The environment must also give internal teams enough visibility and control to operate those pipelines without recreating dependence on an external provider.
The target state combines lower external spend with greater control
Because the engagement remains in progress, the following are target conditions rather than realized outcomes.
If the migration reaches those targets, the financial benefit will be straightforward: the organization expects to remove the approximately hundreds of thousands annual ingestion contract.
The operational benefit is broader. Internalizing the platform is intended to reduce dependency on a third party for pipeline changes, monitoring, troubleshooting, and deployment while giving the company more direct visibility into the systems responsible for delivering data downstream.
Those benefits should be treated as expected outcomes until the migration is complete and the company has validated them in production.
Ownership changes the operating model, not only the technology
The important architectural change in this engagement is not the substitution of one ingestion tool for another.
The company is moving responsibility for a critical data function into an environment its own teams can inspect and operate. That changes how failures can be investigated, how ingestion logic can evolve, and who controls the deployment process.
For enterprises evaluating a similar transition, the decision should therefore extend beyond whether an internal platform can reproduce the pipelines a vendor runs today. The more consequential question is whether the new environment gives internal teams enough observability, reproducibility, and operational control to own the system after migration.
That is the standard Factored is designing toward in this engagement: an ingestion platform that does not simply replace an external service, but can become an internally operated part of the company’s data infrastructure.
