Designing LLM Serving for a B2B SaaS with 100k DAUs
Develop an LLM serving architecture for a B2B SaaS with 100k daily users. The challenge: balancing cost, latency, and model complexity.
The scenario
A B2B SaaS platform is integrating a Qwen3.8-2.4T-A95B model to enhance customer service with natural language processing. The platform serves 100,000 daily active users, with peak concurrent usage reaching 5,000 users. Cost efficiency is crucial due to a tight annual budget of $500,000 for AI operations. Compliance with data privacy regulations is mandatory, requiring robust PII detection and handling.
The system must maintain a latency of under 1 second for 95% of requests to ensure a smooth user experience. Failure to meet this latency could degrade user satisfaction and impact customer retention. With a small team of five engineers, the platform must prioritize operational simplicity to manage the complex LLM deployment effectively.
Requirements
- R1Support 100k daily active users
- R2Maintain p95 latency < 1s
- R3Operate within a $500k annual budget
- R4Handle peak load of 5,000 concurrent users
- R5Ensure PII compliance in data handling
The design
3.1Model Hosting
Amazon SageMaker HyperPod is chosen for model hosting due to its support for model caching, reducing cold start times significantly. This is critical given the 2.4-trillion-parameter size of Qwen3.8-2.4T-A95B. The use of NVFP4 quantization minimizes the memory footprint while preserving model accuracy, crucial for cost management and performance.
3.2Inference and Routing
Inference requests are processed by a pool of vLLM instances on SageMaker, using the model's OpenAI-compatible endpoint. An adaptive model routing strategy is applied to balance load and optimize inference costs, selecting model variants based on the request context and priority. This approach is essential for staying within budget constraints.
3.3Data Privacy and Compliance
AWS's model-agnostic PII detection tool is integrated to ensure compliance with data privacy standards. This tool's ability to adapt to new entity types without retraining reduces the operational burden. It effectively detects sensitive information, ensuring that user data is handled appropriately.
Capacity math
| Metric | Value |
|---|---|
| Daily Request Volume | 100,000 requests |
| Peak Concurrent Users | 5,000 users |
| Annual Budget | $500,000 |
| Latency SLO | p95 < 1s |
Trade-offs
Model Hosting
Picked
SageMaker HyperPod
Passed on
On-prem GPU cluster
HyperPod offers model caching and scalability with reduced cold starts, fitting budget constraints.
Inference Framework
Picked
vLLM
Passed on
TensorFlow Serving
vLLM's compatibility with OpenAI endpoints and quantization support better align with cost and performance needs.
PII Detection
Picked
AWS tool
Passed on
Custom-built solution
AWS's tool reduces development time and adapts to new entity types, crucial for compliance.
Failure modes
| What breaks | Symptom | Mitigation |
|---|---|---|
| Model Cache Miss | Increased latency | Pre-load models during low-traffic periods using SageMaker HyperPod. |
| Routing Misconfiguration | Unbalanced load | Implement adaptive routing with real-time monitoring to adjust model selection dynamically. |
| PII Detection Failure | Compliance breach | Regularly update detection algorithms and conduct audits to ensure tool effectiveness. |
Editor's Verdict
This design prioritizes cost efficiency and operational simplicity, using Amazon SageMaker HyperPod's model caching to minimize latency and costs. At 10x scale, more advanced model routing strategies and potentially hybrid cloud deployments may be necessary to manage costs effectively. Most teams underestimate the complexity of integrating adaptive model routing, which can lead to inefficiencies if not managed properly. Regular audits of PII handling processes are critical to maintaining compliance and trust.
Related LLM Serving coverage
The next blueprint lands Tuesday.
Plus the week's AI/ML data engineering news, curated. Free.