What the engine can be checked against
Matching an input is calibration. Comparing an unused output needs the same definition, population and period, or an explicit assumption that makes the gap acceptable.
What we did about the gaps tonight
The engine now exports quantities that can be put next to public figures. Two pre-registered checks miss: app daily-to-monthly users is 0.21 against a 0.24 to 0.31 reference band, and saves per monthly user are 6.3 a month against at least 10.2 if Pinterest's 2023 weekly save figure still holds. Both rest on unresolved definitions, so neither counts as an independent test.
We also held out each of the four public calibration targets in turn and selected worlds on the other three. With each acceptance gate withheld, three target medians stayed within tolerance: weekly to monthly users, US and Canada growth, and global growth. Outbound clicks per user missed (1.66 against 2.7, plus or minus 0.9). These targets informed model development earlier, so this tests how much world selection depends on each gate, not validity on new data; method in research/25-leave-one-out.md. A backtest against past quarters found rest-of-world users growing too slowly in the engine. A proposed regional growth prior, fitted only to figures published before each start date, cuts the typical miss from 3.8% to 1.6%; a simple seasonal rule still does better at 1.1%. The prior is opt-in until approved (research/24-engine-backtest-fix.md). A full 192-world batch with it passes a pre-stated promotion rule: every kept world meets the original checks, regional growth reaches its targets, rankings within each scenario group barely move (Kendall tau 0.96 to 1.00), only 3 of 402 headline readings flip sign, all within noise, and the three bets keep their readings (web/batch-compare.json, docs/promotion-runbook.md).
We then raised activity until the engine reached those levels and reran all 67 scenarios and the three registered bet checks under that one change. The three decision readings on Bets did not change. Some feature and outside-event rankings did, and seven readings near zero flipped sign, so single readings near zero should not be trusted. A live audit of 360 Pinterest results is on Bets too. Method: research/21-grounding-round.md and research/23-reality-gap-sensitivity.md.
The saved scorecard separates reconciled totals, calibration, prior evidence and independent comparisons. Read the data and source links. The existing tests below answer different questions.