Method comparison

Every Madjik metric can be computed in more than one way: classical code, AI, the language models Claude, ChatGPT and Gemini, a quantum simulator and a real quantum computer. This page explains how the ways are compared, so that a difference between two values can be traced to its cause. The quantum methods are compared step by step on frozen inputs (below); the language models are compared with the classical value on the same inputs, further down.

The steps (quantum methods)

For each compared metric we freeze one set of input data and compute the metric on it step by step. Each step changes one thing.

Step	What it isolates
Classical, exact	The metric's classical definition computed exactly on the same frozen inputs.
Production value	The value Madjik serves, taken nearest in time; not computed from the frozen inputs.
Quantum simulator, ideal	The full quantum circuit with no noise and unlimited shots. A gap to the classical value is the algorithm's own bias.
Quantum simulator, ideal (hardware-sized circuit)	The shallower circuit that fits today's hardware, still noise-free. A gap to the full circuit is the cost of shrinking it.
Quantum simulator, sampled	Finite shots on a simulator. A gap to the ideal value is shot noise.
Noisy device model	The same circuits on a software model of an IBM chip's noise. It predicts what hardware will add.
Real quantum computer (IBM)	The same circuits on a real chip. The plus/minus is shot noise only, not device drift.

Every sampled value carries a standard error (resampling the measured counts), so a gap can be judged against its own noise. Forward-looking metrics are also scored against what actually happened (Brier score, VaR exceedances, AUC).

Compared metrics

M10028 M10036 M10072 M10121 M10122 M50006 M50008 M50020 M50022 M50037

How it scales

Synthetic problems of growing size with a known exact answer (not live metrics), run on simulators. "Runs on IBM today" means at most about 120 two-qubit gates after compiling for an IBM chip, the most that gives a usable result now.

Test	Size	Qubits	Two-qubit gates	Runs on IBM today	Error, simulated shots	Error, noisy device model
Concentration (collision test)	4 categories	4	2	yes	1.0%	1.5%
Concentration (collision test)	8 categories	6	11	yes	1.4%	2.1%
Concentration (collision test)	16 categories	8	31	yes	2.3%	10.5%
Concentration (collision test)	32 categories	10	84	yes	1.2%	10.6%
Concentration (collision test)	64 categories	12	196	no	2.2%	19.7%
Amplitude estimation	4 bins	3	83	yes	0.1%	7.0%
Amplitude estimation	8 bins	4	239	no	0.1%	2.7%
Amplitude estimation	16 bins	5	597	no	0.1%	—
Amplitude estimation	32 bins	6	1,404	no	0.1%	—

Reading it: the concentration test stays runnable up to about 32 categories and its error grows with size. Amplitude estimation is far more precise (about 0.1%) but its circuits get too deep for today's hardware after 4 bins. That depth is what error-corrected machines are expected to remove.

Claude, ChatGPT and Gemini against the classical value

Three language models are calculation methods too. Once a day each model is given a metric's definition and the same input rows BBL's classical code reads, applies the definition itself and returns its own value. The table shows how far each is from BBL's classical value at that moment: the relative difference |model − classical| / |classical|, for the latest run and as the median of the last 7 days. 0.00% means the same value. No metric values are shown. A difference can be the model's arithmetic or its filtering, and also data that arrived between BBL's calculation and the model's read, which matters most for fast-moving inputs. This is a pilot on 14 public metrics.

Snapshot at site build; this table refreshes itself in the browser. Metrics catalog

Experimental and honest: at today's problem sizes the quantum methods are expected to match the classical value within noise, not beat it. Hardware values come from IBM Quantum (Open plan) on a weekly schedule, so there are only a few paired points so far.