Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components

High availability is non-negotiable for production AI systems, especially when compliance is at stake. Salesforce's approach using Amazon SageMaker's SchedulingConfig to distribute model copies across multiple Availability Zones is a credible strategy for teams under similar constraints. It’s a practical solution that leverages multi-model co-hosting to maintain cost efficiency, but here’s the thing: how does it hold up under heavy load? The blog lacks details on the cost implications of scaling this setup, which is crucial for teams contemplating this route.

What they're not saying is that while this solution meets compliance requirements, the actual performance in a real-world scenario with various loads and model types is still an open question. If you’re heavily reliant on AWS and already invested in SageMaker, this implementation could save you from the headache of downtime. However, if you’re considering alternatives like Google AI Platform or Azure Machine Learning, it’s worth comparing their multi-AZ capabilities side by side before committing.

To be clear: this isn't a one-size-fits-all solution. Teams must consider their specific infrastructure and workload patterns. If you’re looking for a way to ensure high availability while keeping costs in check, and you’re already on SageMaker, this could be a fit. But if you’re not on AWS, or if you’re still grappling with data quality issues, it might be wise to hold off.

In the end, I’d recommend putting this on your evaluation list but approach it with a healthy dose of skepticism about its performance at scale. Monitor how others are implementing similar strategies and be prepared to pivot if necessary. This is one tool in the toolbox, not the answer to every high availability challenge.