The Disk Storage Era

We're in the memory storage phase of LLMs when we really should be solving for disk storage. Even Sam Altman's latest interview on Lex Fridman wrongly talks about context windows, because it assumes language models will be able to immediately solve most use cases in mere seconds, with a small amount of context in a single shot. This just isn't how the world works in practice and it's not what people meant by scaling laws.

If you want to build a unicorn, build an orchestrator SaaS. Ideally, it can pick data sources & tools up, use them in focused ways to combine data and accomplish tasks, then put them down and move on to subsequent tasks. Even once it has initially solved a task, it needs to verify its results and ensure they are optimal and somewhat future proof / resilient to truly be valuable.

But to build an orchestrator, we need to focus on solving beyond simple context windows. Context windows are working memory whereas orchestration enables disk storage. Computers today have infinite cloud storage and very large hard drives while working memory is just used to optimize compute.

Currently, LLMs would rather hallucinate than admit they don't know the answer—like that annoying person we've all met at a party. We need to be training systems and consumers that inference-time solutions are required that take their time and optimize for truthiness.

Context window size is helpful, but it's not meant to be elastic in the same way as human cognition where we search, refine, purge, inject, solve, evolve, repeat, onwards and onwards. LLMs are not an elastic computer; they are a real-time interpreter.

Teams like Cognition Labs & Julius have finally started to get the world excited about orchestration again (something Viv did ~8+ years ago with dynamic planning) by enabling this sort of disk storage based approach.

Zapier is well positioned for orchestration, but even they will need completely new interfaces as well as an aggregator of aggregators + orchestrator of orchestrators to reach internet scale and enable all of the world's data and tools to be combined in simple, useful ways.

I've yet to see someone building this orchestration / planning layer in a holistic way from first principles, but the time is now.

LLMs are not foie gras; we can't keep stuffing data down their throats in the hopes of something amazing.

-- Rob