Serving LLM Inference for a FinTech App with 100k Monthly Users
Design an LLM inference system for a FinTech app with 100k users. Focus on latency, cost, and reliability using NVIDIA and AWS tools.
The scenario
A FinTech company is launching an AI-driven personal finance assistant feature, serving 100,000 active monthly users. The assistant provides financial advice via a chat interface, requiring low-latency responses and high availability. The team consists of six engineers with a budget of $50,000/month. Regulatory compliance requires data encryption and audit logging.
If the system fails, user trust could be severely damaged, leading to high churn rates. Additionally, financial advice must be accurate and timely, or it risks regulatory scrutiny. The company aims for a 99.9% availability and a maximum response time of 1 second per query.
Requirements
- R1Handle 200 QPS at peak times
- R2p99 latency under 1 second
- R3Budget cap at $50,000/month
- R499.9% system availability
- R5Data encryption and audit logging
- R6Scalable to 2x current load
The design
3.1Model Serving
We will deploy the LLM using NVIDIA TensorRT Model Connect for its optimization capabilities and ease of deployment. Given the early maturity of this tool, extensive testing will be necessary to ensure stability. The models will be hosted on AWS EC2 instances using NVIDIA A100 GPUs to balance performance and cost.
3.2Load Distribution
AWS SageMaker with Multi-AZ support will manage model hosting to ensure high availability. By distributing model instances across different availability zones, we can achieve compliance with the 99.9% availability requirement. SageMaker's SchedulingConfig will help in efficiently managing resources and costs.
3.3Monitoring and Observability
Elastic Observability integrated with OpenTelemetry will be used to monitor latency, availability, and quality. This setup will allow us to establish SLOs and set up burn rate alerts to preemptively address performance issues. This will involve some operational overhead but is manageable within our team size.
3.4Data Security
AWS KMS will handle data encryption, ensuring all user data in transit and at rest is encrypted. Audit logging will be integrated into the application layer to record all user interactions for compliance purposes.
Capacity math
| Metric | Value |
|---|---|
| Peak QPS | 200 req/s |
| GPU Cost | $25,000/month |
| Storage Cost | $2,000/month |
| Observability Cost | $1,500/month |
Trade-offs
Inference Optimization
Picked
TensorRT Model Connect
Passed on
Custom Model Optimization
Reduces complexity and time to deploy, despite early maturity concerns.
High Availability
Picked
AWS SageMaker Multi-AZ
Passed on
Single AZ Deployment
Ensures compliance with availability SLOs.
Monitoring Stack
Picked
Elastic with OpenTelemetry
Passed on
Custom Monitoring Solution
Provides deep insights with manageable overhead.
Failure modes
| What breaks | Symptom | Mitigation |
|---|---|---|
| Model Latency Spikes | Increased p99 latency alerts | Use Elastic alerts to trigger auto-scaling in SageMaker. |
| Data Breach | Unauthorized access logs | KMS encryption and regular audits. |
| Service Downtime | High error rates in telemetry | Deploy Shadow Engine Recovery for fast recovery. |
Editor's Verdict
This design opts for proven tools like NVIDIA TensorRT and AWS SageMaker to balance performance and cost, with a focus on meeting stringent latency and availability requirements. While the design is intentionally conservative to ensure reliability, scaling to 10x users will necessitate a reevaluation of GPU resources and possibly adopting newer, more cost-effective hardware.
Many teams underestimate the operational overhead of integrating observability tools like Elastic and OpenTelemetry, but this investment is crucial for maintaining SLOs. The most common pitfall is neglecting to plan for data security and compliance from the outset, which this design addresses through AWS KMS and audit logging.
Related LLM Serving coverage
- 2026-W36Three SLOs every search team needs: monitoring search latency, availability and quality with OpenTelemetry
- 2026-W36Deploy an Open Model from Checkpoint to Inference in Two Commands with NVIDIA TensorRT Model Connect
- 2026-W36Restore LLM Inference Capacity in Seconds with Shadow Engine Recovery in NVIDIA Dynamo
- 2026-W36Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components
The next blueprint lands Tuesday.
Plus the week's AI/ML data engineering news, curated. Free.