Can two simulated universes be made to agree on every galaxy property at once?

§1Question

The cosmic star formation study found that the cosmic star formation history alone cannot separate IllustrisTNG from SIMBA: retuning four feedback parameters brings the two within the scatter between seeds. One observable constrains few combinations of four parameters. Real calibrations use many observables at once, and two models that agree on one can disagree on another.

For a code, the reachable set is every galaxy-population summary its four parameters can produce. The structural distance between two codes is the distance between their reachable sets, in units of the noise that is left when everything else is fixed (1 is the standard deviation of the difference between one run of each). Both codes were run in the same boxes at the same resolution and are read from catalogues made by the same Subfind code with the same field definitions, so no aperture, IMF or resolution difference enters. What remains is the difference between the codes: their subgrid models, and also their hydrodynamic solvers, star formation algorithms, yields and wind treatment.

Five questions, each with a rule fixed in advance: is the distance for all observables with one parameter pair per code structural (J1); which observables are structural even when fitted alone (J2); how much does insisting on one parameter pair cost (J3); which observables carry the separation (J4); and does it change with redshift (J5).

§2Data

Simulations

2,054 CAMELS simulations in 25 Mpc/h boxes (256³ dark matter particles plus 256³ gas elements): per code, 1,000 Latin-hypercube simulations (four feedback factors and two cosmological parameters, a different seed each) and 27 seeds at the fiducial parameters. The feedback factors of the two codes are not the same physical quantities.

Observables

Binned summaries of each simulation's Subfind catalogue at z = 0, 0.95, 2.15, 3 and 4 (data/ensembles/camels-galaxies.json). The observables are the stellar mass function, quenched fraction, specific SFR of star-forming galaxies, black hole mass and occupation, gas and stellar metallicity, half-mass radius and gas fraction, all in bins of 0.25 dex in stellar mass from 10⁹ M_☉. The others are the stellar-to-halo mass ratio of centrals, the halo mass function (a negative control) and the stellar mass and SFR densities. 186 of 530 outputs are eligible, in 39 blocks; few bins survive at z ≥ 3.

Noise

The standard deviation of the 27 cosmic-variance seeds of each code plus the emulators' cross-validation error (median seed noise 0.038 (IllustrisTNG) and 0.035 (SIMBA)); every output is divided by it, and every block of outputs gets the same weight.

§3Result

J1 · All observables at once

With one parameter pair per code, IllustrisTNG and SIMBA stay 4.77 noise units apart over 169 outputs in 35 blocks (11 families at z = 0.0, 0.95, 2.15, 3.0, 4.0): the half-sample median of the distance with the emulators' error at the minimiser removed (5th to 95th percentile 4.64 to 4.86; uncorrected median 4.84); on all simulations 4.82 corrected, 4.85 uncorrected, 6.35 at the fiducial parameters. Same-code controls: at most 0.00 corrected. A distance of 1 is the standard deviation of the difference between one run of each code. With the search range cut to 75% of the half-width the joint verdict is structural. By the rule fixed in advance (amendment A1) the joint verdict is structural. Largest shares of the squared distance: bhocc at z = 2.15 (28%); mzr at z = 2.15 (13%); bhocc at z = 0.95 (10%).

Where the squared joint distance comes from (at the minimising pair; share of the weighted sum of squares, and the rms difference of the block in noise units). The joint distance is a root mean over blocks, so one block far off can leave it small:

J2 · Each observable alone

Fitted alone, 2 of 11 families are structural: black hole occupation, gas metallicity at fixed M*; 6 are removable: stellar mass function, quenched fraction, black hole mass at fixed M*, stellar half-mass radius at fixed M*, gas fraction at fixed M*, stellar mass and SFR density; 3 are unresolved: star-forming main sequence, stellar metallicity at fixed M*, stellar-to-halo mass ratio.

J3 · The cost of agreeing

With one parameter pair for everything the squared distance is 2.82 times what it would be if every family were tuned separately (half-sample 5th to 95th 2.60 to 3.08); for two halves of one code, which is what noise and the extra parameters alone give, the same ratio is 2.04 (95th percentile 2.74). The structural joint distance is classed irreducible within the CAMELS ranges (at least one family is structural even alone).

J4 · What separates the codes

In forward selection the families come in the order bhocc, mzr, size, bhms, fgas, shmr, zstar, sfms, qf, gsmf, rho; the first prefix that is structural has 1 family (bhocc, D = 9.49). That is the same as J2 for that family.

Joint distance without each family (point estimate, half-sample median, verdict):

J5 · Redshift

What the dials do

Change of each family's outputs (rms over the family's blocks, in noise units) when one dial moves across its whole range at the others' fiducial values. Descriptive; the dials of the two codes are not the same physical quantity.

§4Reading

What retuning does. 6 of the 11 families come within the margin of agreement when the four parameters of each code are tuned for that family alone: black hole mass, stellar mass function, stellar half-mass radius, gas fraction, quenched fraction and stellar mass and SFR density. Among them are the stellar mass function, the quenched fraction and the stellar mass and SFR density, the quantities that galaxy-formation models are usually calibrated on, so agreement on those says little about whether two codes are alike (the cosmic star formation study reached the same conclusion for the star formation history).

What it does not. Black hole occupation (11.8 at the fiducial parameters, 9.4 after retuning) and gas metallicity (12.4 at the fiducial parameters, 1.7 after retuning) stay beyond the margin however the dials are set within the CAMELS ranges, so they are what separates these two codes by construction rather than by calibration. Stellar-to-halo mass ratio, stellar metallicity and star-forming main sequence are unresolved, at the margin. With the search kept to 75% of the parameter range the verdict changes for stellar-to-halo mass ratio (unresolved to structural), stellar metallicity (unresolved to structural) and stellar half-mass radius (removable to structural): agreement there leans on the edge of the design, where the emulators are least constrained.

Not one observable. Leaving out any single family leaves the joint distance between 3.7 and 5.2 noise units; without the black hole occupation and mass, whose difference is a seeding choice that no dial changes, it is 3.8 (point estimate, all simulations). The separation is therefore spread over many observables and not carried by one.

The cost of one parameter pair. Requiring one parameter pair for all families instead of one per family multiplies the squared distance by 2.82 (half-sample median); two halves of one code give 2.04 (95th percentile 2.74), which is what noise and the extra freedom alone produce. By the rule fixed in advance the ratio does not exceed the null, so the data do not show that the families conflict beyond what is expected; what keeps the codes apart is the families that are structural by themselves.

Redshift. The retuned distance is 2.8 at z = 0 (68 outputs), 4.2 at z = 0.95 (61 outputs), 6.9 at z = 2.15 (32 outputs), increasing with redshift; at higher redshift fewer and only more massive bins have enough galaxies in the 25 Mpc/h boxes, so the trend is a statement about those bins.

What tells these codes apart. The observables that separate the two codes are those that stay apart after retuning: here black hole occupation and gas metallicity, and with less margin stellar-to-halo mass ratio, stellar metallicity and star-forming main sequence. The conventional calibration targets do not. The dials page evaluates the emulators behind these distances in the browser, so any dial setting can be tried against every observable at once.

What this is not. A statement about two simulation codes at CAMELS resolution and in Subfind subhaloes, not about the Universe: no comparison with observations enters, and an observable that retuning cannot reconcile between two codes is not thereby a better or worse test of either.

The distances in the units of the observables

A noise unit is not the same size for every observable. The table gives, for each family, how large one noise unit is (the rms of the noise scale over its outputs) and the rms difference between the two codes' emulated outputs (equal weight per block) at the fiducial parameters, at the best setting for that family alone and at the best joint setting. Descriptive: read from the emulators at the minimisers of the registered analysis; it is not part of the registered result. The joint setting trades families against each other, so a family is usually further apart there than at its own best.

§5Validation

Validations. All five hold.

Sensitivity analyses (point distance of all observables; none was used to choose):

Deviations from the protocol:

• none

§6Checks made after the run

These were made once, after the registered result existed, and reported whatever they gave. None is part of the registered result and none changes a verdict. They are the checks a sceptical reader would ask for first. The decisions that went into the headline, and what happened when each was changed, are in CHOICES-joint-reachability; objections and answers in SCEPTIC.

§7Limits

• Two codes, and more than subgrid form. The result is about this pair, not about structural uncertainty in general; what separates them includes the hydrodynamic solver, the star formation algorithm and the yields, not only the feedback prescriptions. Black hole occupation and mass compare seeding prescriptions (a discrete model choice that no dial changes), so a structural verdict there is expected by construction (what that means, with the sources quoted); the sensitivity analyses give the result without them.
• CAMELS resolution. 25 Mpc/h boxes, 256³ elements: few galaxies above 10¹¹ M_☉, a mass limit of 10⁹ M_☉, almost nothing at z ≥ 3; star formation and black hole growth depend on resolution in both codes.
• The reachable set depends on the prior, and the minimiser sits at its edge. The four factors span the ranges the CAMELS team chose, not the ranges the code authors would accept; the search reaches the corners of the design, where the emulators are least constrained and noisier than their cross-validation error says. The distances are therefore corrected for the emulators' variance at the minimiser (estimated in every split), and the verdicts with the range cut to 75% are reported.
• The distance depends on the observables and weights. Equal weight per block; the margin of 1 is the scatter between two seeds, a unit and not a physical tolerance. Per-family distances are in the table so that other weights can be read off.
• Subfind subhaloes are not galaxies on the sky. No aperture, selection or survey effect is modelled; that is what makes the comparison clean and also what keeps it from being an observational test.

§8Reproduce

python3 scripts/ingest_camels_galaxies.py        # about 35 minutes; range reads of the public catalogues (needs fsspec, aiohttp)
python3 scripts/analyze_joint_reachability.py    # about 15 minutes; writes data/analysis/joint-reachability.json and the report
python3 scripts/power_joint_reachability.py      # calibration on synthetic worlds
python3 scripts/test_joint_reachability.py       # machinery tests and the stale-output check
python3 scripts/export_retune_data.py            # the emulators of the dials page
node scripts/build-site.mjs                      # regenerates this page from the JSON