Compare LLM Tools Fairly with Matched Tasks

Three tabs open. A leaderboard screenshot you saved last week. A nagging feeling that you still don't know which one to actually use.

That feeling is correct, and it is not your fault. A leaderboard answers "which model scored highest on someone else's test?" Your real question is different: "which tool handles my work, under my constraints, at a risk I can live with?" Those are not the same question, and no ranking site can answer the second one for you.

So we are going to run a small experiment instead. By the end, you will have a handful of real tasks, a filled comparison table, and a decision you can defend — plus the honest limits of what that decision proves.

Why Leaderboards Can't Answer Your Question

A benchmark score is a result under agreed test conditions. That phrase matters. When a model tops a public leaderboard, it did so on a specific set of prompts, scored by a specific method, at a specific moment. Your task is a different set of conditions. The score transfers only partially.

Composite indexes make this worse. Many comparison sites blend reasoning, math, coding, and general knowledge into a single "quality" number. That is genuinely useful for one job: building a shortlist. It is misleading for a decision, because it hides where a model is strong and where it falls apart. A model can rank third overall and still be the worst choice for your specific work.

Here is the reframe I want you to carry through this article: a comparison is not a ranking. It is a controlled experiment on your tasks. Leaderboards narrow the field. Your own matched-task test settles the tie.

One honest boundary up front: this is a small, personal test. It is not a statistically rigorous benchmark, and it will not produce a universal winner. It will produce a defensible choice for your situation — which is the only thing you actually needed.

What You Need Before You Start

Keep this short, because the setup is not the hard part.

• Two to four candidates. More candidates multiply your work without improving the decision. Three is a comfortable number.
• The same interface type for each. Compare chat UI against chat UI, or API against API. Mixing a chat window against an API call introduces a confound you will misread as a model difference.
• A place to record results. A spreadsheet or a plain text table. One row per task, one column per candidate.
• A budget decision. Free tiers, rate limits, and paid pricing differ. Decide up front how much you are willing to spend on the test itself.

If you have not narrowed a shortlist yet, start with a criteria-based tool choice first, then come back here to test the finalists. This article assumes you already have candidates in hand.

Step 1: Write Down the Job Before the Tool

This is the highest-leverage step, and it is the one beginners skip. They open two tools, type "summarize this," and call the result a comparison. That is not a comparison. That is two unrelated events.

Start with one sentence describing the real job: what you produce, for whom, and what "good enough" means. For example: "I turn long meeting transcripts into a short action list for my team, and good enough means every owner and deadline is captured correctly."

Then list the two or three task types that dominate that job — summarizing a long document, extracting structured fields, rewriting for tone, answering questions over your own notes.

Now pick 5 to 10 concrete cases. Real inputs, not invented ones. If your job is summarizing transcripts, use actual transcripts. Invented examples produce results that do not transfer to your real work.

Before you run anything, write explicit success criteria for each case. Do this now, while you have not seen any output. If you write the criteria after seeing the answers, you will unconsciously bend them toward the tool you already like.

Include at least one case you expect to be hard, and one edge case: a missing field, an ambiguous instruction, or a very long input.

Tip: Your success criteria should be checkable by someone else. "Sounds good" is not a criterion. "Captures all five action items with correct owners" is.

Step 2: Fix the Conditions

"Matched" means the candidates face the same test. In practice:

• Same prompt text. Copy and paste the identical wording. Do not paraphrase between candidates.
• Same input data. The same transcript, the same document, the same question.
• Same order of tasks. Run task 1 against all candidates, then task 2 against all candidates. This spreads any fatigue or environment drift evenly.
• Same settings where exposed. Temperature, maximum output length, custom instructions, and whether web search or tools are enabled.
• Same conversation state. Start a fresh chat for every task. If candidate A sees three earlier turns and candidate B starts clean, you are measuring conversation history, not the model.

Strict matching is not always possible. Some products do not expose temperature. Some bundle a system prompt you cannot see. When settings are not comparable, hold the logical contract fixed instead of forcing identical parameters that mean different things in each product. Same job, same input, same expected output shape.

Then write down what you could not match. Unmatched conditions are the first thing to check when a result looks surprising.

Step 3: Run the Test and Record What You See

Run a smoke pass first: one easy case against every candidate. This catches the boring failures — wrong model selected, broken access, a feature that turns out not to be available on your plan. Fix those before you spend real effort.

Then run the full task set once per candidate. Paste raw outputs into your table before judging anything. Judging while you run invites you to stop early on the candidate you already prefer.

Here is a table shape that earns its columns:

Score quality on a small fixed scale, and write one sentence justifying each score. That sentence is what makes the table useful a month from now, when you have forgotten why you gave a 4.

Here is what that looks like in practice. Say your criterion for T1 is "captures all five action items with correct owners." Candidate A returns four items with correct owners and one item with the wrong owner. You score it 4 and write: "Four of five items correct; one owner misattributed." Candidate B returns five items but invents a deadline that was never in the transcript. You score it 2 and write: "All items present but fabricated a deadline — a correctness failure, not a completeness one." Now the scores are not vibes. They are traceable to a criterion you wrote before you saw either answer, and a reader can disagree with your reasoning instead of just your number.

The whole loop looks like this:

define tasks → fix conditions → run per candidate → record → interpret → decide

Keep that loop in view. Every step exists to protect the next one.

Step 4: Read the Table Without Fooling Yourself

LLM outputs vary between runs. The same prompt can produce different answers on two attempts. So a single run per task is a signal, not a verdict.

When two candidates disagree, ask in order:

1. Was the answer actually wrong?
2. Was my task underspecified?
3. Was my scoring rule wrong?
4. Was the feature unsupported?
5. Did the model just vary?

These questions do not all point to the same conclusion. The first two are about the answer itself: if the output is genuinely wrong, or if your task was too vague to judge, that is a real quality difference or a real test defect — and you need to know which. The third is a test defect: your scoring rule was wrong, so fix the rule and re-score. The fourth and fifth are candidate differences, not test problems. An unsupported feature is a genuine capability gap for your use case. Variation across repeated runs is a real consistency difference. Record both as findings about the candidate, not as reasons to distrust your test.

Re-run the cases where candidates disagreed. If candidate B gets it right on the second attempt, you learned something about consistency, not just correctness. That is a candidate difference worth keeping.

Look for a failure map rather than a winner: which candidate fails on which task type, and how badly. Then weight by consequence. A rare failure on a task you run daily matters far more than a frequent failure on a task you run once a month.

Common mistake: Declaring a winner from one run per task. That measures luck, not capability.

Practical Constraints the Table Should Also Hold

Raw answer quality rarely decides the choice on its own. These constraints usually do:

• Cost shape. Subscription versus per-token pricing. A cheap-per-call tool can get expensive fast at volume; a subscription can be wasteful at low volume.
• Latency and rate limits. A slow or throttled tool can lose to a slightly weaker but responsive one.
• Data handling. What you are allowed to paste in, retention settings, and whether the tool is approved for the material you work with.
• Feature gaps that break a workflow. File upload, structured output, tool calling, context length, language support.
• Product churn. Models, prices, limits, and interfaces change. Date your results and expect to re-test after a major update.

Add these as columns or as notes beside your table. A tool that wins on quality but cannot accept your file format has not won anything.

Make a Qualified Choice

Now write the decision as a sentence with conditions:

"For this task set, under these constraints, candidate A is the better default."

Name the runner-up and the condition that would flip the decision. That condition is your decision boundary — the thing you would watch for. If your volume triples, or if data handling rules tighten, the answer may change. Writing it down now saves you from re-litigating the whole comparison later.

Say plainly what the test does not establish. It is not a universal ranking. It does not predict performance on tasks you did not test. It reflects a moment in time.

If the candidates are effectively tied on quality, let the practical constraints decide. Do not run more trials hoping for a cleaner answer. More trials on a tie usually just produce a more expensive tie.

Common Mistakes That Break the Comparison

• Testing with invented examples instead of your real inputs.
• Changing the prompt between candidates and calling the difference a model difference.
• Letting one candidate see earlier conversation turns while the other starts fresh.
• Judging outputs after seeing them, with criteria invented to fit a preferred tool.
• Declaring a winner from one run per task.
• Comparing a chat interface against an API call and attributing the interface's extra features to the model.

Each of these has the same shape: an unmatched condition quietly masquerading as a quality difference.

Extend the Exercise

Once the basic loop works, one modification deepens the skill considerably: run the same task set a second time and measure how often each candidate changes its answer. Consistency is a quality dimension, and it is invisible in a single pass.

Two other extensions worth trying:

• Prompt-variation pass. Same task, two phrasings. Does the candidate's quality hold, or does it depend on lucky wording?
• Structured-output task. Extract fields into JSON, where correctness is checkable rather than a matter of taste.

Then turn the task set into a saved regression checklist. Re-run it when a tool updates, a price changes, or a new requirement appears, and compare against your saved results.

The Decision Rule

Pick the candidate that wins on your real tasks under your real constraints. Write down the condition that would change your mind. Keep the task set.

That last part is the compounding move. The first comparison costs an afternoon. The second one, run against a saved task set and saved results, costs minutes. You stop being someone who reads rankings and starts being someone who runs a small, repeatable check whenever the tools shift underneath you.

Your next step: pick your two or three candidates, write five real cases with success criteria before you open any of them, and run the smoke pass today. The table will tell you more than the leaderboard ever did.