How it works

The simulator compares a made-up population with and without a change. Its results show what follows from the stated assumptions and which unknowns matter most.

Loading the build status.

How a result is made

Five steps, the same for every feature and every outside event. The pictures are schematic: they carry no result data.

  1. Draw a worldPick each unknown once from its plausible range, such as how often people return or how much ads annoy them.
  2. Check it against PinterestWarm up the population, then check its baseline against selected published figures. Keep only worlds that pass.
  3. change switches onas things arewith the changeRun it twiceStart with the same people and shared random seeds. Run once as things are, and once with a feature or outside event switched on.
  4. % changesimulation noise, shown beside itCompareMeasure the percent change between the two runs over the same window. Show the simulation noise too.
  5. no changeRepeatAcross kept worlds, the changes form a range. That range is the result under these assumptions.

The parts of the system

Research defines the inputs. The engine makes the runs. The site reads stored results and can run one world live.

Research tables

Journeys, features, outside events and input ranges. Each has a source or is marked as assumed.

research/
World file

Build tools join the research, place allocations and engine assumptions into one file.

world/world.toml
Engine

Simulated people start projects, choose apps and move through sessions. A change acts on a named step.

src/engine.rs
Batch runner

Draw worlds, check baselines, run both versions and collect their changes.

src/main.rs
Results and site

Show the spread across worlds, the input assumptions and live runs.

src/server.rs

Written cast, beside the pipeline. A language model writes names, life sketches and interviews for the People pages. The engine supplies the diaries. tools/make_cast.py

Two ways to model a feature
  • Research priors. The feature reaches a share of sessions and lifts the chance that a session moves the task forward, by amounts taken from the research ranges. These ranges are assumptions informed by published analogues.
  • Mechanism. The feature changes how a session plays out (for example, refinements stop getting dropped), and the engine works out the consequence. This says what the model's structure implies.

Agreement gives some comfort, but both methods could be wrong together. Disagreement shows where the answer depends on the model.

Where a language model is used

A decision model scores how well each feature suits each journey and stage. Only the relative pattern is used: it moves a feature's reach between journeys and leaves the total alone. It never sets how big an effect is. No model is called while the simulation runs.

Inside one simulated day
  • Projects start. Each person may start a project in one of the journeys, at a rate set by the journey mix.
  • Work happens somewhere. On a day they work on it, the visit goes to Pinterest or to another app. The split depends on the journey, the person's habit and how well recent visits went. A rival's launch shifts this split.
  • A session plays out step by step. Each step shows pins and ads. A step is useful or it isn't. Refinements can get dropped when the person moves between results, a pin and related pins. Useful steps lead to saves, click-outs and progress to the next stage: exploring, collecting, comparing, deciding.
  • Frustration ends sessions. Irrelevant results, products when ideas were wanted, unwanted AI images and heavy ads all add to it.
  • Habit moves. Good sessions raise the pull to come back, poor ones lower it, and long-idle people drift away. Non-users arrive in proportion to how common Pinterest already is around them, and more readily when a friend has just had a good visit.
  • Seasons. Each journey has peak months, so the mix shifts through the year: gifts and holiday planning toward December, gardens in spring.
  • Money. Ad impressions times price, by region. Each scenario is read under three assumptions about advertisers: fixed price per impression, fixed budgets, and budgets that follow click-outs.
Checks built into every batch
  • Placebo. A change that does nothing must show exactly zero.
  • Known direction. More ads must raise ad impressions and lower engagement, as measured on other platforms.
  • Noise. Every result is also computed on two halves of the population. The gap between them is the simulation noise.
  • Published patterns. A world must match the weekly-to-monthly active ratio, user growth and outbound clicks per user to be kept. Sessions per active day and implied ad price are shown against outside estimates. They are not used to keep or drop worlds, but the ranges behind them were chosen with those estimates in view.
  • What it depends on. For each scenario, the assumptions the outcome moves with most across worlds.

The build stages below record which checks have run and what remains to be tested.

File map
  • src/engine.rs agents, the daily loop and the session model
  • src/config.rs world file and sampling one world from its ranges
  • src/main.rs many-worlds runs, reconciliation, results
  • src/server.rs web server and live streams
  • tools/build_world.py research tables to world file
  • tools/build_regions.py public population tables to places
  • tools/build_register.py research tables to the SQLite evidence register
  • tools/react.py decision-model scoring, cached
  • tools/make_cast.py names and life sketches for the cast
  • tools/refresh.py public sources to the review inbox
  • tools/backtest.py naive forecasting rules against reported history
  • bench/ local against hosted model benchmark
  • research/06-reviews.md what each second-model review changed
  • research/ methods, calibration data, journeys and features, population data

Checks that could embarrass the model

Each of these compares the simulator with something it was not fitted to. They are shown whatever they say.

Scale against reported history

These are median errors from simple rules applied to reported user counts and revenue. They give context for reading small changes in aggregate figures. They do not measure feature effects or tell us what an experiment could detect.

The engine against its own past

Starts the world at a past quarter using only what was published by then, runs a year forward, and compares simulated monthly users with what Pinterest later reported.

Loading.

Simple rules, one year ahead

Typical miss when a simple rule is applied to the reported series.

Loading comparisons.

Build stages

Done means the listed implementation exists. Evidence, validation and coverage gaps remain; what is left under each stage needs real people, data only Pinterest has, or a person's review.

Loading stages.

Known limits

Matching public figures constrains the baseline. It does not validate simulated people or feature effects.

  1. Loading limits.

Public data refresh

A refresh pulls public sources into a review inbox. New figures are checked by hand before they enter the research tables, because sources define the same words differently.

Loading the review inbox.