When AI Discovers the Next Transformer — Robert Lange

Signal: Treat LLMs as mutation engines inside an evolutionary search, not single-shot solvers. Sakana’s Shinka Evolve shows you can discover competitive programs with far fewer evaluations by combining: constrained code diffs/rewrite/crossover, semantic novelty filtering, a global “scratchpad” of insights, and a UCB-style bandit to pick among multiple LLMs per step.

What changed: Instead of betting on one frontier model or prompt, you orchestrate a portfolio of models and operators, and adaptively route mutations to the model that has empirically improved similar parents. This improves sample efficiency and keeps diversity alive long enough to find stepping stones. It also formalizes provenance via an archive/tree of candidates you can audit and reuse.

Who’s affected: Teams building internal solvers (heuristics, search, compilers, optimizers), agentic coding tools, or scientific workflows. The approach shifts investment from “prompt craft” to evaluators, sandboxes, and scheduling—plus vendor diversity. It is particularly relevant when correctness can be automatically verified.

Tradeoffs and gaps: Results remain seed-sensitive and stochastic; starting from strong baselines can trap you in local optima, while weaker seeds improve novelty but need longer runs. Verification is the bottleneck—LLMs do soft checks; you must build hard evaluators to avoid reward hacking. Co-evolving problems and solutions (surrogates, curricula) is acknowledged as necessary but largely unsolved. Multi-file, repo-scale evolution is still awkward, and knowledge diffusion vs isolation is a real tuning problem.

Adoption challenges: You’ll need a distributed job system, evaluator sandboxes, cost controls, and artifact archives. Bandit routing today is non-contextual; richer context-aware selection could outperform UCB but adds complexity. Evidence is promising yet narrow (e.g., circle packing); claims about discovering radical new architectures remain speculative.

What most will miss: Efficiency, not raw capability, is the unlock. Once unit costs drop, you can scale runs, expand seeds, and harvest stepping stones across problems—turning ad-hoc chats into a persistent research pipeline.