Evaluating LLMs performances for your AI task: presenting my LLMEvaluator.
LLMEvaluator is a Windows application for systematically comparing Azure OpenAI models across custom tasks by measuring response time, token consumption, costs, and ROI indicators. It helps organizations select the best AI model for their specific needs by running concurrent evaluations and providing comprehensive performance metrics.
post Stefano Demiliani
"Remember that there's rarely one best model, but the best model depends heavily on your specific priorities and constraints."
Key points
- Choose evaluation metrics that fit your own use case and use public benchmarks only to shortlist models
- LLMEvaluator runs one prompt against several Azure OpenAI models in parallel, using semaphore-based throttling
- Metrics include response time, token usage, throughput, token efficiency, context utilization, cost estimate and compression ratio
- A 1 to 10 ROI score weights response time 45%, cost 45% and output tokens 10%, and ranks the top three models
- Instances, models and per-million-token prices are set in appsettings.json, and .NET 9.0 or later is required
Quotes
- “Remember that there’s rarely one best model, but the best model depends heavily on your specific priorities and constraints.” (Explains that model selection is contextual and depends on individual business tradeoffs between speed, cost, and performance)
Context
- Features: concurrent model evaluation, response time measurement, token consumption tracking, cost estimation, ROI calculation, throughput analysis, context utilization, detailed metrics comparison
- Versions: .NET 9.0
Source: Stefano Demiliani, community blog. Summary, key points and quotes are derived (CONTENT-NOTICE.md); read the original for the full text.