---
id: post/demiliani-com/12779
type: post
title: "Evaluating LLMs performances for your AI task: presenting my LLMEvaluator."
summary: LLMEvaluator is a Windows application for systematically comparing Azure OpenAI models across custom tasks by measuring response time, token consumption, costs, and ROI indicators. It helps organizations select the best AI model for their specific needs by running concurrent evaluations and providing comprehensive performance metrics.
tier: community
language: en
tags:
  - ai evaluation
  - llm comparison
  - azure openai
  - performance metrics
  - cost analysis
  - model selection
  - benchmarking
  - tool
system: copilot
review:
  state: reviewed
  by: opus
  at: "2026-10-08T01:58:16.448Z"
  flags: []
generated:
  at: "2026-10-08T01:26:53.141Z"
  pipeline: 0.2.0
  prompts:
    extract-post: 1
  input_hash: 4759c384a2e0a0c6270e6e480095014be4365b2f84cb3d89a90fc542e79d5e3c
evidence:
  - kind: blog
    url: https://demiliani.com/2025/11/17/evaluating-llms-performances-for-your-ai-task-presenting-my-llmevaluator/
    title: "Evaluating LLMs performances for your AI task: presenting my LLMEvaluator."
    date: "2025-11-17"
    commit: null
    t: null
    quote: null
  - kind: blog
    url: https://demiliani.com/2025/11/17/evaluating-llms-performances-for-your-ai-task-presenting-my-llmevaluator/
    title: "Evaluating LLMs performances for your AI task: presenting my LLMEvaluator."
    date: "2025-11-17"
    commit: null
    t: null
    quote: Remember that there's rarely one best model, but the best model depends heavily on your specific priorities and constraints.
links:
  learn: []
  objects: []
  features: []
  topics: []
  localizations: []
  videos: []
  posts: []
  guidelines: []
post_id: "12779"
source_id: demiliani-com
source_name: Stefano Demiliani
url: https://demiliani.com/2025/11/17/evaluating-llms-performances-for-your-ai-task-presenting-my-llmevaluator/
published_at: "2025-11-17T08:03:54.000Z"
author: Stefano Demiliani
full_text: false
words: 1164
quotes:
  - text: Remember that there's rarely one best model, but the best model depends heavily on your specific priorities and constraints.
    why_it_matters: Explains that model selection is contextual and depends on individual business tradeoffs between speed, cost, and performance
code_objects_mentioned: []
systems:
  - copilot
  - development
versions_mentioned:
  - .NET 9.0
preview:
  embeddable: true
  frame_url: https://demiliani.com/2025/11/17/evaluating-llms-performances-for-your-ai-task-presenting-my-llmevaluator/embed/
  image: https://demiliani.com/wp-content/uploads/2025/11/llmperformanceevaluator_00.png
  image_alt: null
  image_w: 886
  image_h: 330
  site_name: Stefano Demiliani
  favicon: https://demiliani.com/wp-content/uploads/2022/09/sd_wordpress_logo.png?w=32
  probed_at: "2026-10-07T11:51:21.832Z"
---

# Evaluating LLMs performances for your AI task: presenting my LLMEvaluator.

[Read the post](https://demiliani.com/2025/11/17/evaluating-llms-performances-for-your-ai-task-presenting-my-llmevaluator/) · Stefano Demiliani (Stefano Demiliani, MVP) · 2025-11-17 · 1164 words · tier community · reviewed (checked by Opus)

> LLMEvaluator is a Windows application for systematically comparing Azure OpenAI models across custom tasks by measuring response time, token consumption, costs, and ROI indicators. It helps organizations select the best AI model for their specific needs by running concurrent evaluations and providing comprehensive performance metrics.

## Key points

- Choose evaluation metrics that fit your own use case and use public benchmarks only to shortlist models
- LLMEvaluator runs one prompt against several Azure OpenAI models in parallel, using semaphore-based throttling
- Metrics include response time, token usage, throughput, token efficiency, context utilization, cost estimate and compression ratio
- A 1 to 10 ROI score weights response time 45%, cost 45% and output tokens 10%, and ranks the top three models
- Instances, models and per-million-token prices are set in appsettings.json, and .NET 9.0 or later is required

## Quotes

- "Remember that there's rarely one best model, but the best model depends heavily on your specific priorities and constraints." (Explains that model selection is contextual and depends on individual business tradeoffs between speed, cost, and performance)

## Context

- Features: concurrent model evaluation, response time measurement, token consumption tracking, cost estimation, ROI calculation, throughput analysis, context utilization, detailed metrics comparison
- Versions: .NET 9.0

Source: Stefano Demiliani, community blog. Summary, key points and quotes are derived (CONTENT-NOTICE.md); read the original for the full text.