Study · Definitions

How much of the disagreement between simulations is just definitions?

In plain words

Only 4% of 2,567 pairs of curves from different simulations measure the same thing in the same way.

Many differences between simulated curves are not physics. The papers measured slightly different things: another mass aperture, another way of counting stars, another galaxy selection. Going only by what each source documents, 4% of the 2,567 pairs of curves are measured alike. 13% cannot be judged, because a source does not say what it measured. So a gap you see between two curves may not be physics.

Abstract

Two simulations that plot the same relation often measure different things: a 30 or a 50 kpc stellar mass, a Salpeter or a Chabrier IMF, a 10 or a 100 Myr star formation rate. Without correcting any curve, 2,567 cross-simulation curve pairs are classified by what their documented definitions allow, and the shift caused by one definition is measured inside single runs. 4% of pairs have every needed definition documented and equal; 13% cannot be classified because a source does not say what it measured. What can be compared cleanly is limited by what the sources document. Whether definitions explain the disagreement, the data neither show nor rule out (no epoch test survives correction for multiple tests).

Status: Result · descriptive, no protocol
Analysis: ANALYSIS-definition-budget.md
Script: scripts/analyze_definition_budget.py
Dataset: v0.24.0

Limits in brief. This is a description, not a test: there is no protocol and the curves are reused in many pairs. 55% of the simulation curves were read from figures and no outside expert has checked them. A definition we could not find is not an absence of an effect. Details in §4.

Question

How much of the disagreement between simulations at fixed stellar mass (or halo mass) can be attributed to differences in documented definitions, as opposed to physics or numerics? No curve is corrected: pairs of records from different simulations are classified from their documented definitions alone, and their offsets are compared.

The cleanest measurement of a definition is inside a single run that is published under two of them. Nothing else differs there, so the offset is purely definitional.

Data

Every pair of curves from different simulations, for seven relations at five epochs, is classified from the documented definitions alone:

matched: every needed key is stated and equal
differing: a stated difference, for example 30 versus 50 kpc
unknown: a definition is missing in one of the two records
not comparable: a defining quantity differs, for example two different size definitions

468 of 1636 simulation records had definitions recovered from the source papers, each with section, quote and the checksum of the archive read (evidence ledger). Inside single runs, 78 pairs form 30 groups; population splits (active versus passive) and model variations are excluded.

Result

Fig. 1Cross-simulation record pairs by what the documented definitions allow, 2,567 usable pairs. Filling undocumented values from the papers moved 23% of pairs from “unknown” to “differing”; the share that is fully matched barely moves because simulations rarely use the same aperture.

Fig. 2Shift of the relation when only one definition changes inside a single run. Dot: median over the shared x range and epochs, in the relation’s y units (dex for logarithmic quantities). Bar: range of the per-epoch medians. Zero means the definition does not matter. Largest local offsets are far bigger than the medians, mostly at the bright or massive end: see Table 1.

Table 1 · Definition changes inside single runs (30 groups)
Simulation	Relation	Definition	From → to	z	Median	Epoch medians	Max \|Δ\|
BlueTides	uvlf	dust treatment	dust attenuated → none	8–13	+0.006	−0.002 to +0.088	1.76
COLIBRE	gsmf	stellar mass	100 pkpc → 30 pkpc	0	−0.004	–	0.52
COLIBRE	gsmf	stellar mass	100 pkpc → 50 pkpc	0	−0.001	−0.001 to −0.001	0.42
COLIBRE	gsmf	stellar mass	100 pkpc → total bound subhalo	0	+0.000	–	0.26
COLIBRE	gsmf	stellar mass	30 pkpc → 50 pkpc	0	+0.002	+0.002 to +0.003	0.32
COLIBRE	gsmf	stellar mass	30 pkpc → total bound subhalo	0	+0.004	–	0.65
COLIBRE	gsmf	stellar mass	50 pkpc → total bound subhalo	0	+0.001	+0.001 to +0.001	1.30
EAGLE	gsmf	stellar mass	100 pkpc → 10 pkpc	0.1	−0.018	–	0.64
EAGLE	gsmf	stellar mass	100 pkpc → 30 pkpc	0.1	−0.002	−0.002 to −0.001	0.18
EAGLE	gsmf	stellar mass	100 pkpc → petrosian 2D	0.1	−0.014	–	0.22
EAGLE	gsmf	stellar mass	100 pkpc → total bound subhalo	0.1	+0.000	–	0.15
EAGLE	gsmf	stellar mass	10 pkpc → 30 pkpc	0.1	+0.017	+0.017 to +0.017	0.47
EAGLE	gsmf	stellar mass	10 pkpc → petrosian 2D	0.1	+0.006	–	0.49
EAGLE	gsmf	stellar mass	10 pkpc → total bound subhalo	0.1	+0.017	–	0.69
EAGLE	gsmf	stellar mass	30 pkpc → petrosian 2D	0.1	−0.009	−0.010 to −0.009	0.09
EAGLE	gsmf	stellar mass	30 pkpc → total bound subhalo	0.1	+0.001	+0.000 to +0.002	0.29
EAGLE	gsmf	stellar mass	petrosian 2D → total bound subhalo	0.1	+0.013	–	0.37
EAGLE	shmr	halo mass	BN98 → M200crit	0–5	+0.043	+0.021 to +0.100	0.45
FIREbox	smd	stellar mass	stars in all identified galaxies → stars in galaxies with M*>10^9.3 Msun	2	−0.218	–	0.65
FLARES	uvlf	dust treatment	dust attenuated → none	10–15	+0.032	−0.070 to +0.050	1.77
Illustris	gsmf	stellar mass	2R_half stars → 30 pkpc	0	+0.050	–	0.35
Illustris	gsmf	stellar mass	2R_half stars → total subhalo	0–7	+0.065	−0.012 to +0.120	0.44
Illustris	gsmf	stellar mass	30 pkpc → total subhalo	0	−0.013	–	0.75
Illustris	shmr	stellar mass	2R_half stars → stars within R200c excluding satellites	0–4	+0.130	+0.117 to +0.138	0.25
IllustrisTNG	size	size	3D radius enclosing half of the stellar m… → face on circularized 2D radius enclosing …	0.5–4	−0.030	−0.109 to +0.061	0.23
NIHAO zoom suite	btfr	stellar mass	all baryons within virial radius → stars plus neutral gas	0	−0.350	–	1.62
NewHorizon	gas	gas	cold gas n > 0.1 cm^ 3 and T < 2e4 K with… → cold gas n > 10 cm^ 3 and T < 2e4 K withi…	0.25–2	−0.440	−0.593 to −0.331	1.06
SPHINX20	sfms	SFR timescale (Myr)	10 → 100	4.64–8	−0.165	−0.362 to −0.103	0.91
THESAN-1	uvlf	dust treatment	dust attenuated → dust attenuated	6	+0.001	–	0.98
THESAN-zoom	uvlf	dust treatment	Bouwens+16 empirical attenuation relation… → fiducial THESAN ZOOM dust model	12–14	−0.000	−0.006 to +0.005	0.20

Largest median shift: −0.44 dex (NewHorizon, cold gas fraction, gas). Largest local offset: 1.77 dex (FLARES, UV luminosity function, dust treatment). Median and range are in the relation’s y units (dex for logarithmic quantities), over the shared x range and the epochs listed; Max |Δ| is the largest offset at any single x. None of these is a physics difference.

Fig. 3For each simulation, the share of definition values the comparison rules need that were stated in the original table or figure (dark), recovered from the source paper (vermilion), or are still open (hatched). 432 values remain open: 343 where the paper was read and does not state it, 80 where it has not yet been read for that key, and 9 where the source is a data release without a paper.

A result that went away

Before the definitions were completed from the papers, one stellar-mass-function epoch (z about 6) looked significantly better matched for definition-matched pairs (p = 0.004, recorded in the changelog for 0.17.0). After 468 records were completed it did not (p = 0.13): the apparent detection was produced by undocumented fields. Of 18 epoch tests, 1 fell below p = 0.05, where 0.9 would be expected by chance, and 0 survived Bonferroni correction. The data neither show nor exclude a definitional contribution.

Limits

Digitized records are included. 55% of the simulation curves are digitized from figures and are never rankable; the rankable-only run in the data is the robustness check and is much smaller.
Definition labels are coarse. “Unspecified” hides everything, and two labels that read alike may differ in detail.
Within-run pairs are few. The sensitivity catalogue covers 30 groups; absence of a measurement is not absence of an effect.
Descriptive. There is no protocol; each curve is reused in many pairs, so the tests are not independent. The mass-function study asks a narrower question with a single pre-specified statistic.
Not independently verified. No expert has checked these records yet; a single author confirming a simulation’s definitions would change the status of its curves.

Reproduce

node scripts/annotate-definitions.mjs          # apply recovered definitions (data/definition-fills.json)
python3 scripts/analyze_definition_budget.py   # writes the analysis document and data/analysis/definition-budget.json
node scripts/build-site.mjs                    # regenerates this page from the JSON

Method, parameters, the per-epoch table and every sensitivity pair: ANALYSIS-definition-budget.md.

How much of the disagreement between simulations is just definitions?

§1Question

§2Data

§3Result