Studies
Studies
Analyses that use the dataset to ask a question with an answer. Each has a protocol, a generated analysis document, a stated power and stated limits, and one script regenerates it from the data.
Index
| Study | Abstract | Status | Headline |
|---|---|---|---|
| How much is just definitions? | 2,567 cross-simulation curve pairs are classified from their documented definitions alone: 4% are fully matched and 13% cannot be classified because a source does not state what it measured. 0 of 18 epoch tests survive correction. Descriptive; no protocol. | Result | 4% of 2,567 cross-simulation pairs fully matched |
| Can matching definitions close the gap? | Express the z ≈ 0 stellar mass functions of 9 simulations in one IMF, without added scatter and in a 30 pkpc aperture, and ask what fraction R of the spread across simulations disappears. R = 0.14 (95% interval −0.89 to 0.51; permutation p = 0.08): not detected by the rule fixed in advance. | Pre-registered · result | R = 0.14 (95% CI −0.89 to 0.51), p = 0.08: not detected |
| What can retuning change in cosmic star formation? | Published curves of 9 model families, the authors' own variants and 2,054 CAMELS simulations of IllustrisTNG and SIMBA. Across the 7 families defined in both redshift bands the spread is 0.17 dex at z = 0.5 to 3 and 0.37 dex at z = 5 to 7.5 (funnel not detected). Retuning four feedback parameters per code brings the two codes from 0.22 to 0.08 dex rms (removable). | Pre-registered · result | Funnel not detected, factor 2.2; retuning: 0.22 to 0.08 dex (removable) |
| Can two universes agree on everything at once? | IllustrisTNG against SIMBA over 169 binned galaxy-population outputs at z = 0 to 4: retuning four parameters per code leaves them 4.8 noise units apart at once (margin 1). Alone, 2 of 11 families stay structural (black hole occupation and gas metallicity) and 6 can be matched (including stellar mass function, quenched fraction and stellar mass and SFR density). | Pre-registered · result | Joint D = 4.8 (structural); 2 of 11 families structural alone |
| Does the shape of star formation tell the codes apart? | Does the share of galaxies between the blue cloud and the red sequence, and the width of the star-forming sequence, separate IllustrisTNG from SIMBA in CAMELS beyond what the quenched fraction and the main sequence already do? Over 18 outputs in 4 blocks (z = 0 and 0.95) the shape alone is removable: half-sample median 0.00 noise units, same-code controls at most 0.00, 0.51 at the fiducial parameters. Added to the quenched fraction and the main sequence it adds no difference we can detect (excess over the controls, 5th to 95th percentile -0.21 to -0.10 noise units); added to the joint set of the earlier study it adds no difference we can detect (-0.26 to -0.22). Two of the four checks set in advance did not hold (V1, V4): the emulators of these outputs are less accurate than the run-to-run scatter; the search for the best setting is not stable with four random starts; with up to 64 starts the distance settles at 0.21. The rules were fixed before the data were ingested, with one amendment made before any distance existed. | Pre-registered · result | shape alone removable; added to the usual two: adds nothing detectable; added to everything: adds nothing detectable |
| How much does a model's output change with resolution? | Published curves of one model at two resolutions, from the dataset's own records: 332 pairs of curves could be built, in 17 sets of runs. In the 30 steps between adjacent resolutions with a box stated to be the same (5 sets of runs), the rule fixed in advance gives 2 converged, 28 resolution-dependent and 0 that cannot be told. For the 19 steps with a counterpart in the joint study, the change is at least as large as the distance between IllustrisTNG and SIMBA in 16. Internal check; CAMELS itself is not run at a second resolution. | Pre-registered · result | 28 of 30 same-box steps change more than the CAMELS noise; 16 of 19 are at least the TNG–SIMBA distance |
Not a study but an explanation: how the two codes plant their black holes, the rule behind the property that stays apart in the all-at-once study.
Headline numbers are read from data/analysis/*.json when this page is built. Dataset v0.24.0.
How studies are registered
- Protocol first. A pre-registered study has a protocol committed to the repository before any outcome statistic is computed. The protocol fixes the data, the statistic, the decision rule and the sensitivity analyses.
- What had been looked at. The protocol says what had been examined when it was written (which curves existed, how each simulation documents its definitions), so the reader can judge what the rule could still have been tuned to.
- Amendments are dated. A change after the protocol is committed is an amendment, dated, with a statement of what had been seen when it was made. The protocols of the cosmic star formation study and the all-at-once study were each amended once, before the run, after an internal review (not by an outside expert).
- Generated analysis. The analysis document is written by the analysis script from the data, with every departure from the protocol listed as a deviation. The numbers on the study pages are read from the script’s JSON output, not typed.
- Power. Where a decision rule can be calibrated, its false-positive rate and power are checked on synthetic data before the real data are read (cosmic star formation and all-at-once studies), or reported with the result (mass-function study). The all-at-once calibration uses synthetic worlds with the real block layout whose true distance is known.
- Null results are reported. A rule that is not met is reported as not met, with the interval, and the page says what the test could and could not have seen.
- The definitions study has no protocol. It is a descriptive analysis; its limits motivated the mass-function study.
The dated trail
The commit history shows the order. The table gives, for each pre-registered study, when its protocol was committed, when each amendment was, and when the result file first appeared. The hash is the commit's fingerprint; the amendments say what had been looked at when they were written.
| Study | Protocol committed | Amendments | Result file first committed |
|---|---|---|---|
| Can matching definitions close the gap? | 2026-10-14 0050e60 | 2026-10-14 b7c240b, 2026-10-14 70a9eff | 2026-10-14 d91afcc |
| What can retuning change in cosmic star formation? | 2026-10-14 056d0d8 | 2026-10-14 bdfabc2 | 2026-10-14 670bd75 |
| Can two universes agree on everything at once? | 2026-10-14 955da0a | 2026-10-14 33ca6fe | 2026-10-15 36ada98 |
| Does the shape of star formation tell the codes apart? | 2026-10-23 6e282b1 | 2026-10-23 3bf65ec | 2026-10-23 1d623cf |
| How much does a model's output change with resolution? | 2026-10-23 c452db4 | none | 2026-10-23 4a847ac |
Protocols on the repository: PROTOCOL-harmonization.md (mass-function study), PROTOCOL-model-form.md (cosmic star formation study), PROTOCOL-joint-reachability.md (all-at-once study).