Implementing ML Observability for Fast Fashion's AI Personalization
Design an ML observability system for a fast fashion retailer using real-time data to personalize user experiences. The challenge: ensuring data quality and managing drift at scale.
Data flows from user interactions through Kafka, enabling real-time feature management and model serving, with observability ensured by Prometheus and CloudWatch.
The scenario
A fast fashion retailer with 10 million monthly active users is deploying a personalization engine powered by machine learning to recommend products in real-time. The system handles 500,000 daily recommendations with a latency budget of 500ms per request. The engineering team consists of 5 members, and the system must comply with GDPR regulations. If the observability system fails, data drift could degrade model performance, leading to decreased user engagement and revenue loss.
Requirements
- R1500,000 recommendations/day
- R2Latency SLO: 500ms per request
- R310 million monthly active users
- R4GDPR compliance
- R5Budget: $20,000/month
- R6Drift detection within 24 hours
- R7Data quality monitoring
The design
3.1Data Ingestion
Data ingestion is managed via Apache Kafka, which handles event streams from user interactions. Kafka is chosen for its ability to manage high throughput and low latency, crucial for real-time recommendations. It supports 10,000 messages per second, ensuring we meet peak demand.
3.2Feature Store
Feast is used as the feature store to manage feature data. It works well with Kafka and provides low-latency access to features during model inference, which is critical for our 500ms latency budget. Feast also supports versioning, aiding in compliance and auditing.
3.3Model Serving
We use Seldon Core for model serving within a Kubernetes cluster. Seldon Core is chosen for its scalability and integration with various monitoring tools. Each model pod is limited to 100 QPS to ensure stability and predictability.
3.4Observability and Monitoring
Amazon CloudWatch is employed for infrastructure monitoring, while Prometheus handles application metrics. Drift detection is implemented using Evidently AI, which checks for data distribution shifts and model performance metrics every 24 hours.
3.5Compliance and Security
Data is anonymized and encrypted using AWS KMS before storage to comply with GDPR. Access is controlled through AWS IAM policies, ensuring only authorized personnel can access sensitive data.
Capacity math
| Metric | Value |
|---|---|
| Peak QPS | 220 req/s |
| Kafka Throughput | 10,000 msg/s |
| Seldon Core Capacity | 100 QPS per pod |
| Drift Detection Interval | 24 hours |
Trade-offs
Feature Store Selection
Picked
Feast
Passed on
Hopsworks
Feast's compatibility with Kafka and low-latency access was prioritized over Hopsworks' richer feature set due to latency constraints.
Model Serving Platform
Picked
Seldon Core
Passed on
TensorFlow Serving
Seldon Core's Kubernetes integration allows for easier scaling and monitoring.
Monitoring Tools
Picked
Prometheus
Passed on
Datadog
Prometheus offers a cost-effective, open-source solution that integrates well with Kubernetes.
Failure modes
| What breaks | Symptom | Mitigation |
|---|---|---|
| Data Drift | Decreased recommendation accuracy | Evidently AI detects drift and alerts for model retraining. |
| Kafka Overload | Delayed event processing | Auto-scaling and partition tuning to handle load spikes. |
| Model Latency | Increased response times | Load balancing across Seldon pods to maintain performance. |
Editor's Verdict
This design is deliberately straightforward, focusing on proven tools like Kafka and Seldon Core to ensure reliability. At 10x scale, consider more sophisticated data partitioning strategies and potential microservices architectures to maintain performance. The most common pitfall is underestimating the complexity of drift detection; ensure Evidently AI is well-tuned and integrated with alerting systems.
Related Observability coverage
The next blueprint lands Tuesday.
Plus the week's AI/ML data engineering news, curated. Free.