smevals - a small eval suite for evaluating models, prompts, and harnesses

Here's the thing: the emergence of evaluation frameworks like smevals often promises a silver bullet for understanding model performance. But if you're part of a team that’s knee-deep in deploying models, you know that evaluations are only as good as the metrics behind them. The blog post doesn't dive deep into how smevals measures performance or how its metrics stack up against established players like MLflow and Weights & Biases. That’s a crucial omission for anyone serious about model evaluation.

What they're not saying: this is a prototype. While it might be tempting to grab the latest shiny tool, remember that many prototypes come with a slew of unknowns. You need to consider how this fits into your existing workflow. If you're already using TensorBoard or Neptune.ai, what exactly does smevals bring to the table that those don't? If it can't demonstrate a clear advantage or unique metrics, it risks becoming yet another tool that complicates rather than simplifies.

The catch: I see potential for teams focused on experimentation and needing a lightweight solution to test various model configurations. If you're in a fast-paced environment where rapid iteration is key, then smevals might find a spot in your toolkit. Just don’t expect it to be a comprehensive answer out of the box.

In the end, my position is clear: put smevals on your evaluation list. It’s technically interesting, but you should benchmark it against your current stack before deciding to adopt it. Don't let the buzzwords cloud your judgment; check its performance on your actual data first.