Human behavior simulation is close to usable, but check the 85% claim

The claim. Joon Sung Park, CEO of Simile, says generative agents can predict human behavior with 85 percent plus accuracy, close to the human test-retest ceiling of 86 to 89 percent. The numbers come from a Stanford validation study where agents built from interviews, Big Five personality tests, economic games, and published randomized controlled trials predicted how real participants answered. It is a strong claim with a tidy ceiling, and it is also the founding story of a vendor.

The lineage. Park built the idea from the 2023 Generative Agents paper, the Smallville simulation where 25 GPT-3.5 agents woke up, worked, formed relationships, and organized a Valentine's Day party without scripting. Earlier Social Simulacra work simulated a subreddit to prototype community behavior. The research arc moved from toy town to prediction benchmark, which made the company plausible. The story is coherent, but the leap from emergent fun to reliable measurement is where skepticism belongs.

The product. Simile sells human behavior simulation for concept testing, message testing, and product testing. The company trains population-level and individual-level models using representative panels from Gallup and about 40 vendors, collecting data from tens of thousands of people weekly. Internally, Park describes siblings, agents built from Slack histories that teammates can ask questions like whether he would approve a budget. That is an unusual internal use, and a privacy red flag.

The trade-off. The 85 percent accuracy is not general. It applies to discrete-choice questions in controlled studies, with Pearson correlations around 0.7 to 0.75 for effect sizes. People with heavy web presence, like Park and host Lukas Biewald, are overrepresented in base models; everyone else needs expensive panel data. No independent benchmark verifies Simile's commercial claims. The human ceiling framing is convenient because it makes any score near 85 percent look near-perfect.

What it means. For engineering leaders, the interesting shift is using simulated users to test product concepts, messaging, and designs before shipping, potentially replacing early focus groups and cheapening iteration loops. The operational risk is treating simulation as ground truth. Distribution-level prediction is useful for product decisions, but individual prediction, especially from Slack data, invites consent, privacy, and accountability problems. This is a bounded pilot area, not a core infrastructure bet.

Bottom line. This is a credible vendor pitch with academic roots, not proven production reality. Teams should ask Simile for segment-level accuracy, panel composition, and independent evaluation before spending budget. Watch the space, pilot narrowly, and keep real user feedback in the loop.