No provider influences the numbers
No provider pays for or influences the barometer, and none sees results before they are published. Aprex itself uses models from several of the providers in the solutions we build.
Background
Why a Norwegian barometer?
Foundation
Eight principles behind the test.
- 01
Controlled tasks
Tasks require explicit JSON fields that can be checked. Free writing is not measured in v0.5.
- 02
Key before the run
Answer keys and scoring rules are written before any model answers. The task set is frozen before candidate calls; changes require a new version and a new run. No edits because a model disagrees.
- 03
Equal conditions
The same tasks and declared conditions within each track. Subscriptions and APIs are assessed separately.
- 04
Unpublished task set
12 public tasks are shown with their keys; 12 calibration and 24 holdout tasks are not public and stay off the site, out of the website source and out of metadata. The repo is private, but hidden tasks are not immune to training. Holdout used for tuning is retired with a versioned new holdout.
- 05
Automatic, no judges in the score
Scores come from strict field rules, with no AI judge. Share of correct fields (fieldAccuracy) is diagnostics only and never decides the ranking. Free prose quality is a future separate track.
- 06
Observed rank
Rank shows observed scores, not proven superiority. Stratified paired comparisons on the same tasks are planned. Results are reported per split with holdout-24 as the primary comparison; results from public tasks and calibration are never combined into a claimed measurement on unseen tasks.
- 07
Cost kept apart
Score, API cost and response time sit side by side and are never mixed. Subscriptions report no API cost.
- 08
A person decides
No run is published before a person has reviewed the outliers.
Task set
48 tasks total, six work types.
Each of the six types gets eight tasks: two public, 2 calibration and 4 holdout. Two per type are public with their answer key, so anyone can see what the models get. 12 are available now; the remaining 36 are calibration and holdout, which are not public. Difficulty levels are author-graded (2 easy/3 medium/3 hard per type), not empirically calibrated. The tasks are in Norwegian.
| Types | Document extraction, Customer reply facts, Meeting decisions, Applying supplied rules, Classification, Nynorsk word forms |
|---|---|
| Content | Fictional companies, people, documents and rules. No customer data, not even anonymised. |
| Language | The Nynorsk type is written in Nynorsk; the other types are in Bokmål. |
| Regulation | Supplied fictional rules in each task. Does not test legal expertise and is not legal advice. No expert verification; model agreement never validates a key. |
| Each task has | Task text with rules for unknown, missing or conflicting information, key with evidence per field, an approved alternative, wrong answers including hard negatives, and strict field rules. Tasks are never chosen to favour any particular model or provider. |
| Check | The key and the approved alternative must score 100, and wrong answers at most 0, before a task can be used. Each key is source-checked before the freeze; the revised calibration split is being assessed as part of method development. Nynorsk has only listed variants and does not measure fluency. |
Run conditions
The subscription pilot and planned API track.
| Subscription pilot | Local runner for Codex and Claude Code with existing subscription sign-in. No API keys or pay-as-you-go fallback. First approved measurement pending. |
|---|---|
| Product and model | Product, requested model, reasoning level and CLI version are recorded. Actual model is unknown unless returned by the tool. |
| Planned API track | Vercel AI Gateway with exact model IDs. A separate track, not enabled by the subscription runner. |
| Instructions | «Du løser en arbeidsoppgave for en norsk bedrift. Følg instruksjonene i oppgaven nøyaktig, og svar bare med det oppgaven ber om.» (Norwegian: you solve a work task for a Norwegian business; follow the task exactly and answer only what it asks.) |
| Temperature | Unknown in the subscription pilot. Planned API track: 0 where supported. |
| Reasoning | Requested level is recorded per product and model. This does not mean equal compute across models. |
| Tools and web search | Off |
| Max answer length | Unknown in the subscription pilot. Planned API track: 2000 tokens. |
| Time limit | 120 seconds |
| Repeats | 3 per task |
| Errors and quota | The subscription runner stops and preserves completed answers. Missing answers are listed separately, not automatically scored zero. Resuming is an explicit action. |
| Split | prepare --split public|calibration|holdout|all (default public, holdout needs an explicit split); subscription config taskSplit (actual config: calibration). Calibration is used for method checks and calibration. The primary comparison is the holdout total (24 tasks). Holdout used for tuning is retired with a versioned new holdout. |
| Track | APIs, terminal products and browser chats have different product instructions. We report the surface and do not treat them as the same test. |
| Unpublished tasks | The repo is private, but hidden tasks are not immune to training. Calibration and holdout are not public: they are not shown on the site and are not in the website source or metadata. A runner sends its assigned task text to the provider on each run. |
Scoring
How an answer earns points.
From task to total
How the table numbers are calculated.
| Task score | Mean of the three runs (each 100 or 0; missing is not scored as 0) |
|---|---|
| Type score | Mean of the tasks in the type within one split (primary: 4 holdout tasks) |
| Total score | Mean of the six types within one split. All types count equally. |
| Primary comparison | The holdout total: 24 reserved tasks, 4 per type. Results are reported per split; a pooled mean across all splits is not an independent ranking. |
| Rank | Observed scores, not proven superiority. Stratified paired comparisons on the same tasks are planned. Results from public tasks and calibration are never combined into a claimed measurement on unseen tasks. The local CLI computes no confidence intervals. |
| Stability | Spread between the three runs |
| Cost | API-track actual price per call, including hidden reasoning, scaled to 100 tasks. Given in US dollars excluding VAT, because providers bill in dollars. Subscriptions report no API cost. |
| Response time | Median time from sending the call to receiving the full answer |
Quality control
Nothing is published automatically.
| Before a run | Automatic checks: the self-test of every task and scoring rule must pass. Each key is source-checked before the freeze; the revised calibration split is being assessed as part of method development. |
|---|---|
| Completeness | Every model has answered all 48 tasks in all three runs (12×48×3=1728 answers with the current catalog, which is not fixed). Missing is not full coverage. Results are reported separately for public tasks, calibration and holdout. |
| Error rate | At most 5 percent per model |
| Outlier list | Models that moved more than 15 points since last week, missing or failed calls, and 10 percent randomly chosen answers (173 answers with the current catalog) plus outliers. The pilot measures actual time. |
| Approval | A person at Aprex reviews the outlier list and records approval in the results file. No auto-approval. Final publication is pending; a run counts as approved only with approval recorded in the results file. |
| The website | The site does not build if a run has invalid numbers, a missing work type, missing approval/CSV, wrong version/repeats/coverage, or the protocol is draft with published runs. |
Versions
Numbers are compared only within one task set.
Data
Download and use the numbers.
Every run is published as a CSV file with one row per model, and the barometer page carries structured data (schema.org Dataset). The numbers are free to use with Aprex credited and a link to the barometer.
| Column | Content |
|---|---|
week | Week of the run, e.g. 2026-U41 |
model_id | Model ID in AI Gateway |
model | Model name |
provider | Provider |
tier | topp (top) or rimelig (budget) |
total | Total score 0–100 |
ci_low | Lower bound, interval for future runs |
ci_high | Upper bound, interval for future runs |
dokumentuttrekk | Work type score 0–100 |
kundesvar | Work type score 0–100 |
oppsummering | Work type score 0–100 |
regelverk | Work type score 0–100 |
klassifisering | Work type score 0–100 |
nynorsk | Work type score 0–100 |
cost_per_100_usd | Measured API price for 100 tasks, USD excl. VAT (API track; subscriptions report no API cost) |
median_latency_ms | Median response time in milliseconds |
error_rate | Share of calls that failed after one retry, 0–1 |
Limits
What the barometer cannot tell you.
- Results apply to these 48 controlled tasks, not all Norwegian business needs. A model may be stronger or weaker on your own tasks, and free writing is not measured.
- Strict all-or-nothing gives 0 on one mistake. That is deliberately strict in v0.5 to avoid false partial credit.
- Running through an API with fixed settings is not the same as using ChatGPT, Claude, Gemini or Copilot in the browser, where the provider adds its own instructions and tools. API and subscription are separate tracks.
- Models are updated often, sometimes without a new model ID. A result applies to the product, model ID, CLI version where relevant, and the week it was measured.
- Rank shows observed scores, not proven superiority.
- Difficulty levels are author-graded, not empirically calibrated.
- Temperature 0 makes answers stable but not always identical. That is why each task runs three times. (API track; temperature is unknown in the subscription pilot.)
FAQ
Technical questions
- Who decides the answer key?
- Aprex writes each key before a run and links every field to the task text or a stated language source. The program checks answers against that key. Passing self-tests verifies the checks on those examples; it does not prove that our key is infallible. The task set is frozen before candidate calls; confirmed errors require a new version and recalculation of affected results. No edits because a model disagrees.
- Why temperature 0?
- Temperature 0 is planned for the API track where supported. It does not eliminate all variation. In the subscription pilot we cannot control or verify temperature, so it is recorded as unknown.
- Why three runs per task?
- Answers vary slightly even at temperature 0. Three runs give a more reliable mean and show how stable each model is.
- Why no AI judges in the score?
- AI judges can be wrong and prefer their own style. Fixed rules can be checked by anyone. Free prose quality is a future separate track and is never mixed into the numeric score in v0.5.
- Why is web search off?
- We measure what the model does with the information it is given. With web search, results would depend on the search engine, the timing and which pages exist.
- How are reasoning models handled?
- We record the requested reasoning level per product and model. Identical level names do not imply equal compute, and hidden product settings may be unknown. The subscription pilot does not report measured API cost.
- Can I repeat the test myself?
- Yes, with the 12 public tasks. Task text, answer keys and the full method are open, so you can run them against the same models and compare. Calibration and holdout are not public and are not in the website source or metadata, but the repo is private without immunity to training.
- I found an error. What do I do?
- Write to post@aprex.no. Confirmed errors are fixed in a new version with a new run and recorded in the changelog.
Questions about the method?
Write to us if you want to know more, have ideas for tasks or have found something that should be fixed.
Email Aprex →