Methodology
Scores are deterministic. AI writes editorial prose only. The public site reads a published snapshot through the DAL, never from a live benchmark scrape. When the methodology changes, the update is versioned and applied only through a republished snapshot, so buyers always read the same rules that produced the current roster.
Official PickAI scores
Quality and Value are the only official PickAI scores. They rank the same published roster with different rules, so buyers can compare raw capability separately from buying power.
Quality Score
Official vs raw views
The product has two official scores and two raw benchmark views. Intelligence exposes Humanity's Last Exam in thinking mode with no tools. Coding exposes SWE-Bench Pro directly. Both keep companion benchmarks visible without turning them into extra PickAI scores.
Official scores
Quality ranks capability across the shared benchmark pool. Value ranks the smartest qualified models for the lowest normal-use token cost.
Raw benchmark views
Intelligence and Coding show direct benchmark evidence beside the same roster. They surface supporting signals, but they do not rewrite the official score logic.
What stays separate
Score coverage
Y means the signal is part of the ranking rule. ? means the signal is tracked and shown when available, but the row note defines whether it affects a disclosed component. X means the signal stays out of the scoring formula.
Official PickAI scores
Quality and Value are formula-driven. The score order is public, deterministic, and separate from the benchmark-only tabs.
Quality Score
Quality is capability-first: HLE, SWE-Bench Pro, and OpenRouter operational capability are the only scored inputs. OpenRouter responsiveness is display-only. Price is not a standalone Quality input.
HLE
Primary capability anchor.
SWE-Bench Pro coding
Canonical coding input when an exact current model row is verified.
Supporting buyer signal
PickAI Conversation Value is a proprietary buying-power benchmark. It helps explain subscription value in buyer terms, but it does not change the Quality leaderboard order.
AI tools badges
Badges on the AI tools page are selected through AI editorial review. They are not scores, do not affect model rankings or tool order, and are never assigned because of affiliate payouts. The question-mark badge means the tool has not been reviewed deeply enough yet to assign a more specific signal.
Broad public awareness, strong adoption, or frequent discussion makes this tool especially recognizable.
Trust & governance
This page is public and buyer-facing, but the safeguards behind it are still explicit. Publication rules, update cadence, and operational controls all exist to stop unverified benchmark noise from leaking into the public ranking.
How publication works
Evidence & benchmark sources
Public ranking rows are published only when the underlying benchmark and pricing inputs can be tied to accepted reputable sources that any buyer can fact-check. Unsupported sources are omitted rather than estimated.
Reasoning & novel problem solving
Primary benchmark evidence for raw reasoning views and supporting problem-solving context.
Primary public leaderboard source for ARC-AGI-2.
Primary public leaderboard source for MathArena Expected Performance.
FAQ
Quality ranks the published roster using HLE, SWE-Bench Pro, and OpenRouter operational capability. Responsiveness is display-only. Value asks which qualified model is smartest for the lowest normal-use token cost, using HLE and published API prices.
The Intelligence view is the Humanity's Last Exam lens in thinking mode with no tools over the same published roster. It surfaces HLE evidence directly, while Quality remains the broader deterministic ranking.
SWE-Bench Pro sets the rank, while Terminal-Bench 2.1 and 2.0 stay in separate companion columns so buyers can compare agentic terminal evidence without mixing benchmark versions.
This staged formula version uses fixed normalization ranges established from its reviewed 11-model reference cohort, so the public top-10 cap cannot change scores. Responsiveness is display-only until OpenRouter provides a stable documented performance feed. Price is not applied as a direct Quality bonus or penalty; published token costs affect only the separate Value conversation-affordability calculation. Unsupported capability signals stay display-only until comparable accepted evidence exists.
Value Score
A model must score at least 20 on HLE to qualify. Standard conversation affordability carries more weight than raw HLE, and token costs are scaled so large price gaps count fairly.
OpenRouter operational capability
Context, tools/structured output, and multimodal support.
OpenRouter responsiveness
TTFT and throughput are display-only and never change scores or ranks.
Companion coding benchmarks
Displayed beside Coding when exact-variant evidence is verified; not separate Quality inputs.
Subscription price
Displayed for buyer context; not a standalone Quality input.
Token costs
Can feed disclosed conversation-affordability buyer components when active; never guessed or used as a hidden override.
Value Score
Value ranks qualified models by a plain buyer question: what is the smartest model for the lowest normal-use token cost?
HLE
Required floor and smartness anchor.
Standard conversation cost
Measured with the same normal-use conversation basket for every model.
Subscription price
Shown as context, not scored separately.
Missing pricing
Not guessed. A model needs published input and output token prices for Value.
Raw benchmark views
Intelligence and Coding remain direct benchmark views over the same roster. Companion metrics stay visible, but they do not become hidden score inputs.
Intelligence View
This remains a Humanity's Last Exam view in thinking mode with no tools over the same roster. Supporting benchmarks stay visible, but they do not create a third PickAI score.
HLE (thinking, no tools)
Ranking signal.
MathArena
Displayed as supporting evidence when available.
ARC-AGI-2
Displayed as supporting evidence when available.
AA-Omniscience
Independent factual-reliability evidence displayed when available; never a Quality or Value input.
Context window
Not part of the raw intelligence view.
Speed
Not part of the raw intelligence view.
Price
Not part of the raw intelligence view.
Coding View
This remains a raw SWE-Bench Pro view over the same roster when public SWE-Bench coverage exists. Terminal-Bench 2.1 and 2.0 stay in separate companion columns only when accepted exact-variant coding evidence is available, and they do not create a fourth PickAI score.
SWE-Bench Pro
Ranking signal.
LiveCodeBench Pro
Displayed as supporting evidence only when an accepted exact-variant row is available.
Terminal-Bench 2.1
Displayed separately as supporting agentic terminal evidence only when an accepted exact-variant row is available.
Why the separation matters
Capability stays capability
Quality and Value do the ranking work. Supporting context does not leak into the official order.
Buyer context stays visible
Pricing, app access, benchmark companions, and editorial notes stay available where they help a purchase decision.
Raw views stay raw
Intelligence and Coding expose direct benchmark evidence rather than a blended third or fourth PickAI score. For Intelligence, the ranked HLE row follows the site policy: thinking mode, no tools.
Trust stays explicit
The page tells you which signals rank, which signals inform, and which signals are intentionally excluded.
Public review or reputation signals suggest this tool is especially well regarded.
The tool stands out for a distinctive workflow, product concept, or AI implementation.
The tool is especially useful for creative work such as design, video, images, audio, or writing.
The tool is especially useful for workflows, agents, integrations, or operational automation.
The tool is especially useful for research, source review, search, or evidence-heavy knowledge work.
The tool is especially useful for team, meeting, workplace, or collaboration workflows.
We have not reviewed this tool deeply enough yet to assign a specific badge.
Benchmarks under review
These coding benchmarks test different facets of software capability. SWE-Bench Pro feeds Quality and ranks the Coding view when coverage exists, while Terminal-Bench 2.1 and 2.0 appear in separate columns only when accepted exact-variant companion rows are available.
ARC-AGI-2 tests novel visual pattern reasoning without prior exposure. It will be considered for inclusion only when comparable verified scores exist across all up to 10 models and a methodology version update is published.
ARC-AGI-3 is a watchlist benchmark for interactive agent reasoning. It will not enter the formula until stable comparative data exists across the published roster.
The rules we do not bend
01
Quality Score, Value Score, and benchmark sub-scores are read-only numbers. AI can write editorial prose, but it never changes the score fields.
02
The public site reads from the snapshot through the DAL. It does not fetch benchmark sources or call AI at request time, and unsupported benchmark packs are withheld rather than rendered.
03
There is no combined score. Quality and Value remain the only official PickAI scores, while available raw benchmark tabs stay direct views over the same roster.
04
Public pages and endpoints are capped at 10 models. The published roster tracks the latest stable up to 10 models in the current snapshot, with no overflow pagination or hidden extra rows.
05
Any AI-assisted verdict, pros/cons list, or best-for block carries a visible “AI-assisted, editorially reviewed” label and data-ai-generated="true".
When updates happen & what we refuse
Update cadence
Hard exclusions
Primary public leaderboard source for Humanity's Last Exam.
Independent max-reasoning HLE evidence for GPT-5.6 Sol, Terra, and Luna; accepted provisionally as tier-3 evidence.
Independent exact-configuration evaluation used for Grok 4.5 HLE and display-only companion evidence.
Third-party HLE leaderboard reporting Artificial Analysis data. Use exact MiniMax M3 rows only; not a MiniMax-published result.
Coding benchmarks
Primary and corroborating sources for SWE-Bench Pro, Terminal-Bench 2.1, and Terminal-Bench 2.0.
Primary public source for SWE-Bench Pro results when exact current model variants are listed.
Official Terminal-Bench 2.0 leaderboard. Rows are agent-plus-model results on terminal-bench@2.0 and are verified by a Terminal-Bench team member.
Official vendor release with exact Grok 4.5 SWE-Bench Pro and Terminal-Bench 2.1 results.
Third-party coding leaderboard with exact model rows for SWE-Bench Pro and companion coding benchmarks.
Third-party benchmark comparison page with sourced LiveCodeBench Pro rows. Use only exact model-variant values.
Third-party benchmark leaderboard with sourced SWE-Bench Pro rows. Use only exact current model-variant values.
Pricing sources
Published API pricing sources used for Value, disclosed conversation-affordability components, and buyer cost context.
Official Anthropic model availability and API pricing source.
Official Anthropic announcement for Claude Code subscription usage-limit increases and Claude Opus API rate-limit increases.
Official Google pricing source for Gemini API rates.
Official Kimi pricing source for API token rates and membership plans.
Official OpenAI pricing source for API token rates.
Official xAI pricing and model availability source.
Performance sources
Observed TTFT and throughput evidence shown as display-only buyer context; it never changes Quality or Value.
Operational model metadata used by Quality v2 for context length, max completion tokens, supported parameters, and input/output modalities.
Public TTFT and throughput source for the separate responsiveness signal. This is not benchmark truth.
Bounded fallback cross-checks
Reputable third-party fallbacks used only when direct benchmark sources need bounded support.
First-party leaderboard maintained by the benchmark creators. Report the scores as Aider publishes them, using the aider harness and the open-source benchmark dataset on GitHub.
Primary public source for LiveCodeBench methodology and results.
Primary public leaderboard source for SWE-bench Verified.
Independent cross-reference that mirrors the official Aider Polyglot leaderboard. Useful for validation, not as the canonical source.
Official Moonshot AI model-card eval result files for Kimi K2.6. Results are vendor-published and must disclose harness details.
Official GPT-5.5 launch source for vendor-reported benchmark rows, including the self-reported Terminal-Bench 2.0 companion value.
Official GPT-5.5 system-card source for model behavior and evaluation context.
Official GPT-5.6 source for model identity, pricing, specifications, and vendor-reported benchmark rows.
Reputable third-party model evaluation pages used as bounded fallback evidence when benchmark-owner or same-vendor public sources are unavailable.
Third-party coding leaderboard with exact model rows for SWE-Bench Pro and companion coding benchmarks.
Third-party benchmark comparison pages with sourced tables and transparent methodology used when primary pages are unavailable.
Official model-card benchmark tables hosted on Hugging Face and cited when they provide current public benchmark details.
NVIDIA-hosted model card for DeepSeek V4 Pro with model metadata and benchmark table. Treat as accepted tier-3 evidence and cite exact model rows only.
Aggregator comparison page that includes Aider Polyglot. Use only as a tertiary cross-check.
Use the models index and the model detail pages when you want pricing, app access, buyer-facing guidance, and the supporting benchmark context next to the ranking.
Terminal-Bench 2.0
Displayed as supporting agentic terminal evidence only when an accepted exact-variant row is available.
HLE
Not part of the raw coding view.
Context window
Not part of the raw coding view.
Price
Not part of the raw coding view.
