9:00–9:15 AM
Intro & logistics



Tim Hockin, Eitan Yarmush, Ron Lev
Google, Solo.io, NVIDIA
Agent Substrate Day
Monday, November 9, 2026
The community gathering for engineers building infrastructure for AI agents on Kubernetes, from Google, Solo.io, and the Agent Substrate community.
9:00am–5:00pm · Sky SLC, Salt Lake City
Add to calendar Google Outlook Apple
Register on Luma9:00–9:15 AM
Intro & logistics



Google, Solo.io, NVIDIA
9:20–9:30 AM
10 min

Solo.io
Agent Substrate 101: Building Isolated, Governed and Scalable AI
Agents need to be isolated for what they can and can't access (files/folders, production envs, LLMs, MCP tools, runtimes, etc. - think autonomous and standard Agents) and at the same time, scale up and down based on load, if/how the Agent is being used, and how it's being used.
By default, Agents have access to every file, LLM, command, and authN/Z implementation that you do. At the same time, you're running Agents in a scalable, performant, and reliable way with a proven out orchestration/clustering platform.
Example: You want to run 200+ Agents, but only spread across 6-8 k8s Pods. The Agents still perform as well as expected while keeping the underlying infrastructure costs (both resources and actual financial costs) low.
In this intro to Agent Substrate session, you will learn about the OSS project that makes this happen - Agent Substrate, with key topics including:
1. Build on top of k8s primitives (clustering of resources for performant sandboxed Agents).
2. Implement a specialized and efficient a Control/Management Plane that's built for the world of agentic.
3. How Substrate Actors running Agents are 90% more efficient (proven in benchmarks) for cost optimization in comparison to Agents in k8s Pods.
4. Efficient and much smaller footprints for running Agents via Actors.
9:35–9:45 AM
10 min

Solo.io
A platform engineers dilemma: agent-sandbox or agent-substrate?
As developers clamor to build agents, platform engineers responsible for managing Kubernetes are faced with a dilemma: do they run those agents as pods? Do they sandbox them with agent-sandbox? Or do they run them with agent-substrate? In this session we'll review the differences between these options and articulate why you ought to consider agent-substrate and the advantages it brings.
9:50–10:10 AM
20 min

AWS
Running Agent Substrate in Production on EKS with microVMs
Agent Substrate packs many stateful actors onto a small pool of warm worker pods, now runnable on EKS with a microVM sandbox for pods. This talk is about what it takes to run that in production on AWS: which pieces offload to managed services, actor snapshots on Amazon S3 and the control-plane database on RDS or Aurora PostgreSQL, and where the gaps still are.
Those gaps are the point: no AWS credential provider for actor secrets yet (the interface ships Kubernetes Secrets and Google Secret Manager, not Secrets Manager or Parameter Store), no automated idle-suspension or worker-consolidation policy, and cloud-neutral transport and pod-identity work still worth upstreaming. The goal is an honest map of where Agent Substrate on EKS works today and where AWS and the community can push it.
10:15–10:35 AM
20 min
Solo.io
kubectl Is Lying to You: What a Live Visualizer Taught Me About Agent Substrate
Run `kubectl get pods` on an Agent Substrate cluster and you see a few quiet worker pods. Meanwhile untold numbers of agents are restoring from snapshots, queuing for slots, and checkpointing back to object storage. The interesting state isn't in pods. So I built Substrate Scope, an open-source live visualizer that shows the runtime as it really works: worker bays, a restore queue, the snapshot shelf, and agents moving between them in real time. Its buttons act on the real cluster: they scale the WorkerPool, run a demand-driven autoscaler, fire surge traffic, and reset a wedged pool.
The visualizer I built turned out to be the best way to find Substrate's edges and frankly, to explain how it works for those of us who need crayons to understand things (that would be me). On my laptop cluster I ran 18 agents on two worker pods and served over a thousand real sessions, with a minimum reserved capacity 4.5x lower than the same agents as always-on pods. This talk is a live demo plus an accounting and discussion of what broke along the way and how we work through it.
- My tool ended up with lost running sessions because I read the actors list. Every finished session leaves a Suspended actor behind; mine passed through thousands of entries in a day and started returning partial results. The workers list is the source of truth.
- CPU and memory are the wrong autoscaling signals! Workers are slot-bound and active sessions idle at single-digit mCPU. Substrate rejects when the pool is full instead of queuing, so the real demand signal is in your callers' retry loops.
- My autoscaler made things worse functionally worse. On a serial model backend, more workers just piled more requests onto one generation queue. Everything piled up and timed out and it was a hot mess. We'll look at how we address this.
- Killed sessions left ghost actors pinning workers while the pool looked idle. This made me sad but it's fixable.
- "Pods vs. substrate" charts lie if you compare usage. Idle pods use almost no CPU but reserve it forever. Reservation is the number to keep an eye on.
You'll leave knowing what to watch on a Substrate cluster, what to scale on, which failures to expect under load, and a tool you can point at your own cluster today.
10:35–11:00 AM
11:00–11:20 AM
20 min

Solo.io
Network Policy at Substrate Scale
Congratulations, you've got substrate running on your cluster! But after the eureka moment fades, you're now tasked with figuring out how to integrate your shiny new actors into your existing security and compliance posture. Come join us for an exploration of the tools substrate provides to establish a zero trust architecture for agents and how those tools compose other cloud-native-components like service mesh and Kubernetes NetworkPolicy.
11:25–11:55 AM
30 min
12:00–12:05 PM
5 min

Cloud Foundry Foundation
Five Reasons I thought Agent Substrate Would Fail
12:05–12:10 PM
5 min
12:10–12:15 PM
5 min
12:15–12:20 PM
5 min


Truly Sandboxed Networking: Moving beyond MVP
In this session we will discuss the current Substrate network isolation and directions for improving isolation for Actor workloads. What we currently have in the project is closer to a minimal viable infrastructure and this breakout seeks to map out the post-V1 roadmap for the project.
12:20–12:25 PM
5 min

Using Substrate for anything but Agents
Agent Substrate is for agents. It is right there in the name. Or is it? Strip away the LLM orchestration frameworks, and what you are left with is a systems engineer's dream: an ultra-high-density, multi-tenant sandboxing platform built on Kubernetes that bypasses standard Pod lifecycle constraints.
What happens when we hijack these primitives for workloads that have absolutely nothing to do with AI? In this talk, we will explore the limits of Substrate by utilizing its core building blocks: native process-level suspend/resume, rapid rehydration, and pod-decoupled scaling.
Through this exercise of “unconventional systems-engineering design patterns” we’ll look at the good ideas, the terrible ideas, and how pushing this infrastructure to its limits helps us all gain a better understanding of why these exact primitives make Substrate the perfect foundation for the agentic era.
12:25–1:25 PM
1:25–1:30 PM
5 min

Solo.io
Postcard from 2028: A Platform Team’s Retro on Agent Substrate
It’s November 2028. A 10,000 person company has run its agent platform on Substrate for two years. There are 20 agents for every employee, most sessions are started by other agents rather than people, and some cases stay suspended for weeks. This talk is that platform team’s retro: what worked, what hurt, and what they’d tell themselves in 2026.
Each story in the retro starts from a trend we can already see, including power users with a hundred agents, agent-to-agent fan-out, long-running case work, shared agents acting on someone’s behalf, and unattended automation. Each one is followed to where it ends up.
Then we come back to today. Substrate is in alpha, and every one of those stories comes down to a decision that’s still cheap to change. We’ll walk through the personas, use cases, and a concrete list of asks, backed by customer design sessions and measurements from running kagent on Substrate.
1:30–1:35 PM
5 min
1:35–1:40 PM
5 min

NVIDIA
Suspend-safe networking
Suspending an actor today cuts every TCP connection it holds, so an agent waiting on a slow call cannot be suspended.
This session shows a concept design that keeps those connections alive while the actor is suspended and restored on any worker, wakes it when data arrives, and demonstrates it live with an inbound request that waits on an outbound call across a suspend.
1:40–1:45 PM
5 min

Where Do the GPUs Go When Your Agents Run on Agent Substrate?
AgentSubstrate puts agents to sleep when they are idle and saves their memory and files in a snapshot. GPU memory can't be saved this way, so GPU work doesn't fit into the sandbox.
My setup keeps the GPUs outside. kagent agents run on Substrate and send requests to a local model on Ollama. HAMi splits each GPU into slices with memory limits, so many agents can share one GPU without one agent taking all the memory.
I'll show the setup, the numbers I got when many agents woke up at once, and what broke on the first try.
1:45–1:50 PM
5 min

NVIDIA
Where Multiplexing Pays: Scheduling Sandboxes for RL Rollouts
Agentic RL fans each training step into many mostly-idle sandboxes. We ran NeMo Gym’s SWE-bench agent on Agent Substrate (suspend/resume multiplexing during LLM calls) and OpenSandbox (one pod per rollout).
In some cases we were able to lower the wall time by over 30% by suspending actors on Substrate during LLM calls. We’ll walk through how we benchmarked both platforms, and the surprises we hit along the way.
1:55–2:25 PM
30 min


NVIDIA
Scheduling sandboxes like threads: a policy layer for Agent Substrate
Agent Substrate multiplexes many of actors onto a small pool of warm workers, but it does not decide who runs next.
When a pool is full, resume calls to ateapi fail, and the router parks the request and retries, and whichever retry lands first wins the worker.
For reinforcement learning rollouts and coding-agent fleets, where sandboxes outnumber workers by an order of magnitude, that order decides how long a generation step takes and which rollouts make progress at all.
This talk presents an orchestration layer we are building on top of Agent Substrate's public API. It adds what an operating system scheduler gives threads: a resume queue ordered by policy, preemption of idle sandboxes to make room,
a free-worker buffer, and per-pool policies declared as a Kubernetes resource. Underneath, KAI Scheduler places the worker pods themselves, so tenants get hierarchical queues with quota, fair share and reclaim across their WorkerPools.
We will walk through the design and example use cases and configurations for effective multiplexing.
2:30–2:50 PM
20 min


Agent Substrate Environment
When we first revealed Agent Substrate, people were mind blown to see we can suspend and resume an actor at every tool call without impacting user experience. Built on top of Agent Substrate, Agent Substrate Environment is aiming to provide a higher-level abstraction designed to deliver a fast agentic environment for the entire community.
As agents evolve into autonomous problem-solvers, they require secure, stateful, and scalable environments to execute code, manipulate files, and run commands safely. While Agent Substrate supplies the sandbox runtime and orchestration primitives, we developed a higher-level abstraction that focuses on remote file system and process lifecycle operations. This new layer accelerates onboarding and provides a production grade interface for agentic systems that can use remote environments. The environment abstraction also aligns well with the model commonly used by frontier labs, acting as the operational "hands" that execute commands for the "brain."
We will dive into how we built an agentic environment abstraction on Agent Substrate and go into details like actor lifecycle management. Additionally, we will share how this layer helped build multiple production-grade integrations at Google. Ultimately, our goal is to keep this project minimal, modular, and reusable for the broader Agent Substrate community and future ecosystem.
2:50–3:15 PM
3:15–3:35 PM
20 min
3:40–4:25 PM
4:25–4:35 PM
Wrap up



Google, Solo.io, NVIDIA
More agents. Less overhead.
149 Pierpont Ave
Salt Lake City, UT 84101
Tuesday night, after KubeCrawl + CloudNativeFest, the same community keeps going. Details stay on the Luma page.
Luma page