- Tools
- Memory
- State
Agent Orchestration
in Production
Senior AI Engineer, Refik Anadol Studio
- Senior AI Engineer at Refik Anadol Studio
- Building the AI agent for Dataland, a museum of AI arts in Los Angeles
- Before agents: diffusion models, LoRA training, generative art
- Before that: computer vision for manufacturing QC at Bosch
- I write interactive explainers at cobanov.dev
Dataland
The world's first Museum of AI Arts, in Downtown Los Angeles. Visitors talk to a museum agent: artworks, the building, planning a visit.
An agent is a loop
The model picks the next move. The loop around it is ours: what may run, when to stop, what to keep.
Where it breaks
- ToolsLoops that never endSame tool, same arguments, until the timeout.
- ToolsConfident calls to the wrong toolOr the right tool with made-up arguments.
- MemoryContext that grows until it forgetsThe important fact was forty messages ago.
- StateState that lives in one processDeploy, restart, double tap: gone.
Draw the line on purpose
Two questions for every decision: does it need language? What does a wrong answer cost?
The model proposes. Code disposes.
The model fills a typed proposal. Code checks it against reality and decides what actually happens. Rejections go back as instructions.
Scope them to the phase.
Budget them in code.
Tools are prompts
- Few, coarse tools beat many thin ones
- Names and docstrings are instructions
- Return what the model needs to say next, not your table
- Errors are instructions: say what to do instead
Not every tool, every turn
Every tool in the prompt is another wrong option and more tokens. Expose only what makes sense in the current phase.
Budgets live in code
Never ask the model to stop. Make it unable to continue.
- Request, tool call and token limits
- A wall-clock timeout per turn
- Same call, same args: refuse it
- A deterministic fallback answer
Context is not a database
Trimming history keeps you under the token limit. It also deletes facts: silently, by age, not by importance.
Four kinds of memory
Working
This conversation, right now. Lives in the prompt.
Episodic
What happened, and when. Time-stamped events.
Semantic
What is true about them. Facts that can change.
Procedural
How we do things here. Skills, playbooks, tool habits.
Read fast. Write carefully.
Recall is on the hot path, with a latency budget. Remembering happens after the reply, in a worker.
Updating is harder than remembering
The model proposes facts. Code decides what is stored, what is replaced, and what must never be written down.
Keep it out of the process.
Run one turn at a time.
Make the state explicit
The model can suggest the next phase. Code owns the transitions, and the phase decides which tools exist.
State lives outside the process
Any worker can serve any turn. Two turns of the same session never run at the same time.
Takeaways
- 01Let the model decide what needs language. Let code decide what needs guarantees.
- 02Tools are prompts. Design and scope them for the model.
- 03Memory is a write problem before it is a read problem.
- 04State lives outside the model, and outside the process.
Thank you!
memory.cobanov.dev, how agent memory works
kvcache.cobanov.dev, KV cache and attention, interactively