# WeirdML v3 leaderboard

external benchmark

> Every figure below is reproduced as its upstream source published it: nothing is modelled, estimated, interpolated or converted. `Unavailable` means no source published the value — it is never a zero. Each value carries its evidence state and the date it was observed.

Page: https://modelchoice.dev/leaderboards/benchmarks/external-weird-ml-v3

Every model the catalog carries a published WeirdML v3 value for, ranked by that value.

## Benchmark record

| Field | Value |
| --- | --- |
| Category | external |
| Measure | Agent harness runs up to 50M tokens, token-curve scoring |
| Tasks | 11 hand-made ML and data-analysis tasks (15 configurations) |
| Difficulty | Open-ended ML engineering with unfamiliar data and little feedback |
| Published by | [WeirdML v3](https://htihle.github.io/weirdml.html) |

Håvard Tveit Ihle's WeirdML v3 gives coding agents 11 hand-made machine-learning and data-analysis tasks with unfamiliar data, unspecified goals or very limited feedback, and scores how well and how efficiently they improve as they spend tokens.

## WeirdML v3 ranking

No model in the catalog has a published WeirdML v3 score. The benchmark is defined by WeirdML v3, but the catalog carries no value for it yet. An absent value is shown as absent here rather than as a zero.

Rows are ordered by the value the source published, highest first. The source does not state whether a higher value is the better result, so this document does not either: for a benchmark that measures a rate of failure, read the table from the bottom.
