Handling the Volume of Demand.
The ML ad serving platform had limited throughput (50,000 requests per second) and struggled with high latency. Additionally, there was a need for multi-framework support, robust monitoring, and data quality assurance in both the serving and feature store pipeline.
Using ML for Operational Efficiency & Insights.
- We enabled ML inference for multiple model development libraries like PyTorch and TensorFlow using NVIDIA Triton for flexible deployment.
- To enhance reliability, we integrated Prometheus and Grafana for real-time monitoring, improving error handling, automated recovery, and rate limiting to prevent overload.
- Performance was optimized through Grafana load testing, Golang MLServing enhancements, and Aerospike Cache integration.
Scaling Throughput 20X - 1M Per Second.
- Latency as low as 10ms.
- Reduced cost and better performance.
- More reliable and transparent ML operations.
