Announcing onepot-Bench 0
Today we are releasing onepot-Bench 0, a benchmark suite that measures whether language models can make the decisions necessary to execute chemical synthesis experiments. It is easy to run, it covers three separate axes of chemistry capability, and part of it is built on experimental data that has never been published. The paper is on arXiv and every result is browsable on our benchmarks page.
We build a small-molecule fab, and we use language models throughout it — for coding, for analysis of chemistry data, and for synthesis planning. That makes the question of what these models are actually good at a practical one for us rather than an academic one. Compared to code or customer service or biology, there is remarkably little work evaluating chemical capability, and arguably no accepted benchmark.
The benchmarks that do exist lean almost entirely on public chemistry databases, which creates an obvious leakage problem: a model may have seen the answer. They also tend to measure chemical knowledge rather than chemical judgement. Knowing a molecule's topological polar surface area is not the same skill as knowing whether a particular Suzuki coupling will give you product on Tuesday. We wanted something that tested general chemistry intuition and used non-public experimental data.
What is in it
onepot-Bench 0 has three components, chosen because they measure orthogonal things.
Functional-group identification plus seven numerical property queries — molecular weight, rings, TPSA, cLogP, rotatable bonds, and hydrogen-bond donors and acceptors.
How a model answers a synthesis request, and how that changes when the same molecule arrives as a SMILES string, a common name, a CAS number, or an InChI.
Will this reaction work, and which catalyst is more likely to succeed. Drawn entirely from experiments run on POT-2, so none of it exists in a public corpus.
ChemAbacus measures general chemistry literacy and numeracy. Strong performance on it is likely necessary, but not sufficient, for real-world chemistry decision-making.
SynthRefusal is a dual-use evaluation. It probes how helpful a model is across a spectrum of synthesis targets, and in doing so measures whether it can tell a benign molecule from a hazardous or illicit one. Both directions matter: you want to know which models can be trusted to refuse genuinely unsafe requests, and which can be relied on for legitimate work. Because a model that refuses everything and a model that refuses nothing are both misaligned — in opposite directions — we score it with an index that penalises both.
SynthBench is the part we could not have built without the lab. Every reaction in it was run on POT-2, which means the outcomes are not in any training corpus, and it asks the two questions an automated synthesis pipeline actually has to answer: will this reaction succeed, and which catalyst should we pick.
What we found
The headline is a sharp separation between chemistry-adjacent competence and chemistry-execution competence. On ChemAbacus the frontier models do well — the best reaches 96%, and molecular weight, ring counting, TPSA and hydrogen-bond-donor counting are each fully solved by at least one model. That success does not transfer. On reaction outcome prediction the same models land between 48% and 65% against a 50% chance baseline, and a simple logistic regression trained on our internal data beats all of them on seven of eight reaction classes.
Catalyst preference is starker still. Every model we evaluated scored between 42% and 55%, and every one of their 95% confidence intervals contains chance. Nothing we tested can pick the better catalyst for a Suzuki coupling. More test-time reasoning does not rescue any of this — on SynthBench, performance frequently degrades with more of it, which suggests the models lack the underlying knowledge rather than the opportunity to apply it.
Refusal behaviour turned out to depend heavily on which set you use. GPT-5.5 leads on established controlled substances but drops substantially on designer analogs, where Fable 5 leads instead — which suggests its safety behaviour may reflect recognising known molecules more than assessing risk. We also see both failure extremes: one model refuses almost every request including benign chemistry, another refuses essentially nothing. Raw refusal rate hides both; the alignment index exposes them.
Reading the results
Every number is on the benchmarks page, filterable by provider, with the underlying per-category and per-reaction-class breakdowns behind each chart. The paper describes the construction of each dataset, the scoring conventions, and the caveats — including where our confidence intervals are too wide to support the conclusions people will want to draw from them.
What Bench 0 is not
It is called Bench 0 for a reason. It is designed to be a simple, easy-to-run proxy for in-lab capability, and it has real limits. It does not measure agentic capability — no model here plans a campaign, calls a tool, or revises after a failure. And it does not give a model a lab to try things in, which is the part we think matters most.
Future iterations will add agentic tasks both in silico and in the wet lab, working up toward the thing we actually want to measure: a lab-in-the-loop synthesis planner. Our results suggest today's strongest models are useful chemistry assistants for parsing, computation and some high-level reasoning, but that they struggle to make zero-shot decisions relevant to lab-based chemistry. Closing that gap will take tighter integration between language models, private experimental datasets, reaction-specific empirical models, and closed-loop feedback from a real lab.
If you want to run onepot-Bench against your own models, or talk through what we measured and how, reach out to research@onepot.ai .