How much does a model's output change with resolution?

§1Question

The joint study and the shape study both name resolution as their largest open weakness: CAMELS boxes are 25 Mpc/h with 256³ dark matter and 256³ gas elements, and nothing in the project measured how much a model's output changes when only the resolution changes. CAMELS has no second resolution, so that cannot be measured on CAMELS. What can be measured is how published curves of other models change between resolutions of the same model, and how large that is next to two numbers the project already has: the run-to-run noise of the CAMELS observables, and the distance between IllustrisTNG and SIMBA that the joint study reports.

Two questions with rules fixed in advance:

• How many steps are converged? For each step between adjacent resolutions of one model in a box stated to be the same, the root-mean-square change of the curve is measured in noise units (one unit is the scatter between one run of each CAMELS code). Below one unit: converged. With the digitization noise taken out, above one unit: resolution-dependent. Otherwise: cannot tell.
• Is the change smaller or larger than the gap between the codes? The same change, in the same units, set against the IllustrisTNG–SIMBA distance for the same kind of observable.

§2What could be built

A pair needs two records of the same model at two resolutions with the same axes and definitions and a comparable epoch (within 0.1 in redshift). 332 pairs of curves met that rule, grouped into 132 steps in 17 sets of runs. Tier A means the records or the registry state that the box is the same, so a change can be read as resolution (and, where the authors recalibrate at each resolution, as the effect of that recalibration). Tier B means the box, the sample or the run differs or is not stated.

Examined and not built:

• FIREbox, FIRE-2, Romulus25, BlueTides, ASTRID, SPHINX20, FLARES: one resolution only in the dataset.
• Horizon-AGN and NewHorizon: registered as separate families; the records do not state that the model is unchanged.
• NIHAO zoom suite and NIHAO-UHD: different galaxy samples and definitions.
• DARWIN-1 (box) and DARWIN-2 (selected zoom-ins): different galaxy samples and different ids.
• SIMBA m100n1024 and hires (Glowacki+20): parametric records, not evaluated.
• EAGLE colour counts (Ref-L0025N0376, Ref-L0100N1504, Recal-L0025N0752): one point per record, no curve.
• THESAN-SDAO-2: a different dark matter model, not a resolution.
• TNG100 against TNG50 or TNG300; Illustris-2 and -3: no record pairs them with equal definitions.

Other public sources of one model at two resolutions, checked without any account or key:

• IllustrisTNG data pages: 'In order to access any Illustris[TNG] data, you need to first Login.'. Not used.
• EAGLE public database: 'Access to both the database and particle data require only a single account, which you can register for below.'. Not used.
• FLAMINGO: the page links to a separate data portal and lists lightcone particle data and maps; no credential-free galaxy catalogue was found. Not used.
• THESAN: the project page has no data statement. Not used.
• COLIBRE paper data (2509.07960.yml): served without credentials; already in data/raw; the L050, L100 and L200 ladders are read as a supplementary set. Used.

§3Result

• What could be built. 332 pairs of curves of the same model at two resolutions, with equal definitions and a comparable epoch, in 17 sets of runs. In 5 of them (30 steps between adjacent resolutions) the box is stated to be the same. Another 63 pairs of COLIBRE in other boxes come from the public data file of its paper.
• By the rule fixed in advance, of the 30 steps with a common box, 2 are converged (the change is below one noise unit), 28 are resolution-dependent and 0 cannot be told. 19 rest on the CAMELS noise of the matching observable and mass bins; the others on a typical CAMELS noise.
• Against the distance between the codes. 19 of these steps have a counterpart in the joint study. In 16 of them the change between two resolutions is at least as large as the distance between IllustrisTNG and SIMBA for that kind of observable; the median ratio is 2.09.

Steps between adjacent resolutions, tier A, by label.

The change against the distance between IllustrisTNG and SIMBA. R_c is the root-mean-square change in noise units with the digitization noise taken out. The distance is that of the joint study for the same kind of observable: at the fiducial parameters where the authors changed nothing but the resolution, after retuning where they recalibrated. 16 of 19 steps have a ratio of at least 1; the median ratio is 2.09.

The step that starts at CAMELS resolution. The IllustrisTNG and Illustris models in a 25 Mpc/h box with 256³ elements are at CAMELS resolution; the next run has eight times the mass resolution. The observable is the cosmic star formation rate density.

§4Reading it

All tier A steps, with the span from the coarsest to the finest run where there are three. R is the root-mean-square change in noise units, R_c the same with the digitization noise taken out; the last three columns are in dex (a factor of 10; 0.1 dex is a factor of 1.26), or in fraction for the quenched fraction, finer minus coarser. "Noise basis" says whether the unit is the CAMELS noise of the matching observable and mass bins (reference) or a typical CAMELS noise (yardstick), used where the observable has no counterpart or too few mass bins overlap.

Made after the run, to help read it, and deciding nothing: for 7 of 23 steps on mass-binned curves, more than half of the squared change lies in the top quarter of the mass range. For a mass function that is the steep high-mass end, where a small change in mass is a large change in density and a 25 Mpc/h box has few galaxies.

At the margin 0.5 the tier A counts are 0 converged and 30 resolution-dependent; at the margin 2, 12 and 18; without the epochs above z = 2, 2 and 26 (and 2 that cannot be told). Reference noise rebuilt from the joint study's own functions: seed noise reproduced (check passed); the public COLIBRE file is the one the dataset's records were made from (check passed). The typical noise scale used where an observable has no counterpart is 0.065.

Tier B (the box, sample or run differs or is not stated): a small change may be called converged, a large one cannot be put down to resolution. Steps between adjacent resolutions; the last column is the largest root-mean-square change among them, in dex or fraction.

COLIBRE in other boxes (supplementary: read from the public data file of the COLIBRE paper, where the 50, 100 and 200 Mpc boxes are listed at more resolutions than the dataset holds; not added to the dataset).

What the protocol expected before the run, set against the counts:

• Expected: steps whose parameters the authors recalibrated are close. By the rule, 2 of 24 such steps are converged.
• Expected: the code distance exceeds most resolution shifts where the joint study's retuned distance is large, and not where it is small. Of the 2 comparisons with a retuned distance of at least 1 noise unit, 1 have a resolution shift at least as large as the compared distance; of the 17 with a smaller distance, 15.
• Expected: star formation histories change more at high redshift than at low redshift. Not tested separately.

Deviations from the protocol:

• The wording and layout of the report were revised after the run (unit names with the pipe character broke one table; a variant column was added to the comparison table; a section setting the expectations of the protocol against the counts was added); no number was recomputed, and the first output is in the version history.

§5Limits and gaps

• Internal check. No outside expert has reviewed the protocol, the code or the result. The labels are not significance tests: the only uncertainty in R is the digitization of the figures.
• CAMELS is not tested at a second resolution. The changes are those of other models, from their papers. SIMBA has no pair that can be evaluated in the dataset (its published fits at two resolutions are parametric and were not evaluated), so nothing here measures the resolution dependence of SIMBA.
• Different papers, finders and apertures. Each pair compares two curves of one paper. The noise unit comes from CAMELS Subfind catalogues with total bound masses and is only a yardstick for curves with other definitions.
• The curves stop at the authors' resolution limits. Most records keep only what lies above a resolution limit, so the changes below it, where resolution matters most, are not in the data. The numbers understate the effect there.
• Recalibration. COLIBRE and FLAMINGO recalibrate their subgrid parameters at each resolution, so a small change there reflects that recalibration. Only the two Pillepich et al. sets of runs change nothing but the resolution, and they cover one observable, the cosmic star formation rate density.
• One margin. A noise unit is a convention, not a physical tolerance. It is small (typically 0.07 dex), so the labels are strict; the counts at other margins are given above.
• Few independent sets of runs. Epochs and observables within one set of runs are correlated. The counts are counts of rows, not independent tests, and no p-value is given.

§6Reproduce

python3 scripts/analyze_resolution.py       # a few seconds; writes data/analysis/resolution.json and the report
python3 scripts/check_resolution.py         # the checks made after the run
python3 scripts/test_resolution.py          # machinery tests on synthetic ladders and the stale-output check
node scripts/build-site.mjs                 # regenerates this page from the JSON