Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/sec

Running a 1 trillion parameter model locally at 4 tokens per second sounds impressive, but here's the thing: without a complete picture of the hardware configuration, it's hard to take this claim at face value. The use of Intel Optane Persistent Memory is intriguing; it straddles the line between DRAM and SSD. However, this alone doesn't guarantee performance. What they're not saying is whether the rest of the system—CPU, GPU, cooling, and power supply—can sustain such throughput without throttling. Performance benchmarks can be manipulated to highlight specific components while glossing over bottlenecks elsewhere.

The practical implications of this build depend heavily on the specific use case. If you're a data engineer focused on deploying large language models, consider that a couple of tokens per second might not be sufficient for real-time applications. If this setup is indeed a prototype, as suggested, it might not scale well or be reliable under production scenarios. You need to assess whether this build can handle the workload consistently, especially during peak loads.

Who stands to benefit here? Early adopters experimenting with novel hardware configurations might find this build worthwhile. However, remember that the novelty of using Optane PMem does not equate to production readiness. The catch is that unless you're prepared to troubleshoot potential stability and performance issues, you might end up spending more time than it's worth.

In summary, unless you're in a position to validate these claims against your own data and requirements, I'd recommend holding off on this specific configuration. Test it if you have the opportunity, but don't rush to deploy it in a production environment without thorough vetting.