tldw.news
How it works
Every week roughly fifty videos about AI and developer tooling are worth someone's attention, and almost none are worth an hour of it. This is the pipeline that reads them so you don't have to — and the part where a human still says yes before anything is sent.
TL;DR
Six steps, three phases, one human gate.
- CollectMachine
- Curated YouTube channels, polled on a schedule
- Transcripts pulled and stored
- UnderstandMachine
- LLM analysis per video: takeaways and a ranking signal
Corpus
Chunks embedded into PostgreSQL with pgvector, queryable across weeks
- DecideHuman
- A person approves what makes the brief
- The weekly brief goes out
The model ranks and summarises. It never publishes: nothing reaches a subscriber without someone saying yes.
The pipeline
Ingest curated channels
A fixed set of channels is polled on a schedule rather than crawled openly. Curation happens at the source: the corpus is only as good as the list it starts from, and a smaller trusted list beats a larger noisy one.
Fetch and store transcripts
Each new video gets its transcript pulled and stored. Transcripts are the unit of work from here on — the video itself is never the thing being processed.
Analyse each video with an LLM
Analysis runs per video and produces the takeaways plus a ranking signal. The output is a set of candidates, not a published article: nothing reaches a reader at this stage.
Chunk and embed into the corpus
Once a batch completes cleanly, transcripts are split with LlamaIndex and embedded through an OpenAI-compatible endpoint, then written to PostgreSQL with
pgvector. This is what makes the archive queryable across weeks instead of one video at a time — patterns show up in the corpus that no single video makes visible.Indexing runs in a bounded background pass after analysis has been committed, so embedding latency never lands on transcript retrieval or on a reader's request.
A human approves before anything ships
Candidates go to an editorial queue. A person decides what makes the brief. The model ranks and summarises; it does not publish.
The brief goes out
Approved items are packaged into an issue, rendered, and sent. What reaches a subscriber is a short list with takeaways: what is worth watching, and what can be skipped.
Decisions worth explaining
Re-indexing is decided by a fingerprint, not by a human remembering. A source is re-embedded when its transcript hash changes, or when the chunk size, overlap, embedding provider, model, or index version changes. The configuration is part of the stored fingerprint, so a model swap invalidates exactly what it should without anyone bumping a version by hand.
Chunk replacement is atomic. Old vectors stay queryable while new ones are being computed. Only when every embedding for a source is ready does a single transaction swap them. A slow or failed embedding run degrades freshness, never availability.
Batches are bounded and idempotent. Each pass indexes a capped number of sources, and selection is driven by state rather than by a queue position. A cron trigger that fires late, twice, or overlapping is safe to reject and retry.
The human step is not a fallback. It is the point. A generative pipeline that publishes on its own produces volume, and volume is the problem this exists to solve.
Stack
- Backend
- Python, FastAPI
- Database
- PostgreSQL with pgvector
- Chunking
- LlamaIndex SentenceSplitter
- Embeddings
- OpenAI-compatible client on OpenRouter
- Frontend
- Next.js App Router, React, Tailwind
- Scheduling
- GitHub Actions cron
- Agent access
- MCP endpoint over JSON-RPC
The archive is also readable by agents: tldw.news exposes an MCP endpoint over JSON-RPC — described by its server card — and an LLM index, so a coding agent can query the corpus directly instead of scraping the pages.
Who builds this
tldw.news is built and run by Andrea Tarzariol, an engineering manager in Munich. It exists because the tooling landscape moves faster than anyone can watch it, and because the interesting question is not what shipped this week but what it changes for a team on Monday.
Get in touch if you want to talk about any of it.