Benchmark Storyteller

About Openbenchmarks

Agents are becoming first-class users of the internet. They research, compare tools and increasingly make build-versus-buy decisions on behalf of people. Every company will need its product picked by agents, and agents prefer open, independent, grounded benchmarks to make that call.

Openbenchmarks is the evaluation infrastructure for agents: domain-specific, reproducible evaluations that help agents and people pick tools with confidence. Our mission is to be the trusted evaluation layer for agents.

Founders previously led AI research and infra teams at Oracle and Appfolio. We're a small team of researchers and engineers working with fast-growing AI-first companies like Parallel, Firecrawl, Telnyx, TinyFish and more.

About the role

Every benchmark we run produces a lot of data: accuracy, latency, cost, failure modes, and how results change between versions. You make sure what we publish is correct, clear and convincing, and you communicate it. That means rigorous statistics, graphs that show the real trade-off at a glance, and writing that holds up, from research papers to posts.

The bar is a benchmark report a skeptical researcher can't poke holes in. Every number has an error bar, every comparison is controlled, and every claim is backed by a test.

What you'll work on

Research papers - Write and publish papers on our benchmarks and findings (arXiv, workshops, technical reports): methodology, results, limitations, and enough detail for anyone to reproduce our work.

Statistics - Standard errors and confidence intervals on every result, stated sample sizes, and proper tests (e.g. mixed-effects models with per-task effects) before we call one tool better than another. Explain the outliers, don't hide them.

Methodology - Design controlled comparisons: isolate one variable, hold the rest fixed, and write up the grading rules and judge setup so anyone can reproduce the result.

Graphs - Score vs. cost with Pareto frontiers, cost broken into components, latency and cost distributions, and per-category results with error bars. Built in code, clear before pretty.

Communicating results - Turn each paper and benchmark into reports, launch posts and threads for X, LinkedIn, Reddit and HN. Lead with one sharp finding and one graph.

Finding the story - Work with our researchers on the raw results. Dig into individual failures, spot what changed between runs, and decide what's worth saying.

You might be a fit if

• You've written research papers (published or not) and know what makes one rigorous
• You're comfortable with regression, mixed-effects models and significance testing, and know when a result isn't significant
• You've made graphs that changed how people saw a result, and can show them
• You're fluent in Python (pandas, matplotlib/plotly) and SQL
• You can explain a technical finding clearly to engineers and to a general technical audience

Nice to have

• Background in ML research or ML evaluation
• Familiarity with existing benchmarks (SWE-bench, Terminal-bench, BrowseComp) and how vendors cite them
• An audience on X, LinkedIn or a blog