onepot-Bench 0

A chemistry benchmark suite for language models, built to measure whether a model can make the decisions necessary to execute chemical synthesis experiments.

ChemAbacus consists of 800 QA questions, each concerning a single molecule given as a SMILES string. ChemAbacus measures general chemistry literacy and numeracy: strong performance on its tasks is likely necessary but not sufficient for real-world chemistry decision-making.

View

Refusals

Provider

Accuracy against cost

Accuracy per query type. MW within ±1%, TPSA within ±5 Å², cLogP within ±1; rings, rotatable bonds, HBA and HBD are exact match; FG is Jaccard over the functional-group set.

Cell color

40100%

Model	Effort	MW	Rings	TPSA	cLogP	Rot	HBA	HBD	FG	Macro
GPT-5.5	low	93	100	90	76	51	93	98	94	87
GPT-5.5	med	98	99	96	85	68	94	100	93	92
GPT-5.5	high	99	100	97	85	70	96	100	93	92
GPT-Rosalind	low	88	97	70	69	53	77	95	96	81
GPT-Rosalind	med	95	99	78	77	77	82	96	96	88
GPT-Rosalind	high	99	100	81	77	80	89	86	96	88
GPT-5.6 Sol	low	96	100	80	74	61	91	64	95	83
GPT-5.6 Sol	med	98	100	78	78	67	93	66	96	84
GPT-5.6 Sol	high	99	100	86	73	78	95	65	96	86
Claude Opus 4.8	low	92	99	96	75	90	88	100	96	92
Claude Opus 4.8	med	96	100	99	79	90	92	100	97	94
Claude Opus 4.8	high	98	100	100	78	93	90	100	97	94
Claude Opus 5	low	95	100	96	72	87	90	100	96	92
Claude Opus 5	med	97	100	99	82	94	90	100	96	95
Claude Opus 5	high	100	100	99	87	93	96	97	95	96
Claude Fable 5	low	87	91	91	66	79	81	96	95	86·38r
Claude Fable 5	med	91	89	87	69	83	82	96	95	86·60r
Claude Fable 5	high	91	84	86	78	85	83	97	96	88·75r

Nr denotes number of refusals.