Study · Cosmic star formation history

What can retuning change in a simulated universe, and what does its design force?

In plain words

We could not show that simulations disagree more about the early Universe than the later one (the spread is 2.2 times larger, but our test did not pass); retuning four dials per code brings two codes from 0.22 to 0.08 dex apart.

Models of the cosmic star formation rate differ by 0.17 dex at redshift 0.5 to 3, and by 0.37 dex at redshift 5 to 7.5. Our rule, fixed in advance, did not count this as a real widening: the result was "not detected". For two codes, IllustrisTNG and SIMBA, turning their four feedback dials shrinks the difference from 0.22 to 0.08 dex, so by our rule it is removable by retuning. Two cautions: the best match sits at the edge of the dials' range, and one of the checks on the emulators fails narrowly.

Abstract

Simulations are tuned to a few observations at redshift about 0 and then asked what happens elsewhere. We separate what a model can still do by retuning its own parameters from what its form forces whatever the tuning, for the cosmic star formation history. Published curves of 9 model families do not show a detectable funnel of growing disagreement (across the 7 families defined in both redshift bands, the spread is 0.17 dex at z = 0.5 to 3 and 0.37 dex at z = 5 to 7.5: not detected). In 2,054 CAMELS simulations, retuning four feedback parameters per code brings IllustrisTNG and SIMBA from 0.22 to 0.08 dex rms (removable).

Status: Pre-registered · result
Protocol: PROTOCOL-model-form.md
Analysis: ANALYSIS-model-form.md
Power: POWER-model-form.md
Dataset: v0.24.0

Limits in brief. Two codes, one observable and CAMELS resolution. The structural verdicts need a step between the codes of order a dex that the four parameters cannot absorb, so smaller differences are called removable. One validation of the SIMBA emulator misses its threshold narrowly, and the result is read with that qualification (§4).

Question

Two things limit a simulation’s predictions away from the observations it was tuned to: what the model can still do by retuning its own parameters (parametric), and what its form forces whatever the tuning (structural). Climate science separates them with perturbed-physics and multi-model ensembles. Here the cosmic star formation history is the test case.

Four questions, each with a rule fixed in advance: do published simulations disagree more in the early Universe (E1); how big are the authors’ own changes (E2); can retuning remove the difference between two codes (E3); and tuned to the same observed history, do their predictions beyond it stay apart (E4)?

Data

Published: Curves of the cosmic star formation rate density of 9 model families, and the authors’ own resolution, parameter and cosmology variants (data/model-variants.json).
CAMELS: 2,054 simulations in 25 Mpc/h boxes with the IllustrisTNG and SIMBA codes at identical resolution: per code, 1,000 Latin-hypercube simulations (six parameters: Ω_m, σ₈ and four feedback factors, a different seed each) and 27 seeds at the fiducial parameters. Star formation logs reduced to 70 redshift bins (data/ensembles/camels-sfrd.json).

Result

E1 · Do published simulations disagree more in the early Universe?

The spread across 7 families is 0.17 dex at z = 0.5 to 3 and 0.37 dex at z = 5 to 7.5 (a factor 2.2; 90% interval of the log ratio −0.27 to 0.61, Pitman–Morgan p = 0.066): funnel not detected by the rule fixed in advance. It does not survive leaving one family out: without FLAMINGO the sign reverses (log ratio −0.18). FLAMINGO's flagship-resolution run is the lowest curve above z = 5. The same code's run at eight times the mass resolution lies roughly a dex higher at z = 6 to 7, so resolution within one code moves the early star formation history as much as the choice of code (a reading added after the result, not part of the test).

Fig. 1Standard deviation across simulation families of the log star formation rate density at each redshift, with its bootstrap band (16th to 84th percentile of resamples of the families). The number of families changes with redshift; the fall above z = 7.5 is a change in which families are defined there (FLAMINGO’s curve ends at 7.8). The two shaded bands are the ones the pre-specified test compares. Its power was checked on synthetic data: with seven families a funnel of a factor three is detected about two times in three, a factor of two about one time in three, and with no funnel the rule fires in 2% of cases.

E2 · How big are the authors’ own changes?

Within one code, the authors' own resolution and box changes (10 variants of 4 families) move the star formation history by 0.23 dex rms at z = 0.5 to 3 and 0.37 dex at z = 5 to 7.5, against 0.17 and 0.37 dex between the codes. Their parameter variants (IllustrisTNG, 5 runs) move it by 0.48 and 0.34. The sizes are the authors' choices, so only the order of magnitude carries information.

E3 · Can retuning remove the difference between two codes?

At their fiducial parameters IllustrisTNG and SIMBA differ by 0.22 dex rms. Retuning four feedback parameters per code over the same factors brings them to 0.08 dex (emulators trained on half of the simulations: 0.05 to 0.09; the same code against its other half leaves at most 0.01, and two seeds of these codes differ by 0.13): removable by the rule fixed in advance. The best-matching pair sits at the edge of the CAMELS range in 3 of its 8 coordinates, where the emulator is least constrained, and with the ranges cut to a quarter the codes stay 0.13 dex apart. Tuned to the same observed star formation history their predictions overlap at z = 4.1 to 7.7 (the 90% intervals are disjoint at 0% of the redshifts; the medians drift apart by 0.42 dex by z = 7.7): parametric.

IllustrisTNGSIMBAdifference between the codes at fiducial parameters

Fig. 2CAMELS, 25 Mpc/h boxes, the same four feedback factors varied in both codes: the spread the parameters produce in each code (solid), the seed-to-seed scatter of one box (dashed) and the difference between the two codes at their fiducial parameters (ochre). IllustrisTNG vermilion, SIMBA dark. All at CAMELS resolution.

E4 · Tuned to the same history, do the predictions stay apart?

IllustrisTNGSIMBAobserved history

Fig. 3Both codes conditioned on the observed star formation history (Madau and Dickinson 2014, dotted) at redshift 3 and below, with a 0.2 dex systematic common to all points and 0.15 dex for the small box, and what each then predicts beyond it: the model’s mean response, median and 90% interval. Verdict: parametric.

Limits

Validations. Not every validation holds: V4 SIMBA (the SIMBA emulator's cross-validation error is 1.51 times the seed noise at its worst redshift, against a limit of 1.5); the analysis page says which conclusions to read with that qualification.
Scope. Two codes, one observable and CAMELS resolution are all this can speak to, and the parameters of the two codes are not the same physical quantities.
What the structural verdicts can see. Calibrated on synthetic twin codes, they need a step between the codes that the four parameters cannot absorb: about 0.6 dex above z = 3.5 for weak parameter responses and about 1 dex for strong ones. Smaller steps are classed removable or parametric. See POWER-model-form.md.
Resolution and code are confounded in the published curves. The sizes of the authors’ own variants are their choices, so only the order of magnitude of E2 carries information.

The validations, the sensitivity analyses and the full limits are in ANALYSIS-model-form.md; the pre-registration and its amendment are PROTOCOL-model-form.md.

Reproduce

python3 scripts/ingest_camels_sfrd.py   # CAMELS star formation logs to data/ensembles/camels-sfrd.json (public, hashed)
python3 scripts/analyze_model_form.py   # analysis document and data/analysis/model-form.json (about 20 minutes)
python3 scripts/power_model_form.py     # power study, data/analysis/model-form-power.json
python3 scripts/test_model_form.py      # the saved output is not stale
node scripts/build-site.mjs             # regenerates this page from the JSON