AWS Infinity: 75,000 AI Agents on a Raspberry Pi, Zero Compute While They Wait

Most AI agents spend most of their lives doing nothing. A coding agent kicks off a build and waits twenty minutes. A triage agent watches a repository for the next pull request. An assistant waits until morning for a human to approve something. In the standard way of building agents, that waiting still costs money: the process stays alive, the memory stays allocated, and the meter keeps running.

On October 23, 2026, AWS published a different answer. Infinity, released under the Apache 2.0 license as part of the Hydro project, a collaboration between AWS and UC Berkeley, treats a tool call as a pause button. When an agent calls a tool, Infinity dispatches the call, saves the conversation, and suspends the agent entirely. When the result arrives — minutes or days later — the runtime resumes the agent from its saved state. Nothing runs in between.

The headline number from the announcement is striking: AWS says 75,000 concurrent agents fit in the 8 GB of a Raspberry Pi 5. That figure deserves both attention and scepticism, and this article gives it both — along with the architecture, the cost math, and the open questions AWS itself lists.

Quick answer

• What it is: Infinity, an open-source agent runtime written in Rust, released October 23, 2026, by AWS and UC Berkeley under Apache 2.0.
• What it does: It suspends an agent at every tool call, persists its conversation history, and resumes it when a result or event arrives — so a waiting agent occupies no process, no thread, and no running compute.
• The headline claim: In AWS's benchmark, an idle agent costs about 100 KB of resident memory, which puts roughly 75,000 agents inside 8 GB. That benchmark used a simulated model, not a production model API — see the careful reading below.
• What it costs: $0 for the runtime itself. You pay separately for the model API tokens and for the AWS services underneath (Lambda, SQS, Aurora DSQL, DynamoDB), if you deploy on AWS at all.
• Who it is for: Developers building long-running, event-driven, multi-agent systems — not anyone looking for a hosted agent product with an SLA. AWS calls this an early research release.

The problem: waiting is the expensive part

The usual agent harness is a loop: call the model, run a tool, wait for the tool, repeat. While the tool runs, the harness holds a process and its memory, and it cannot react to anything else that happens in the meantime. Multiply that by thousands of agents — one per repository, one per customer ticket, one per long-running job — and the bill is dominated not by thinking, but by waiting.

Infinity's author, Shadaj Laddad, research lead for the Hydro team at AWS, frames the fix with an idea borrowed from programming languages. Languages like Rust, JavaScript and Python let a function suspend at an await and resume later, because the runtime saves the function's continuation — the state it needs to carry on. Infinity applies the same trick to agents: at the moment an agent calls a tool, its conversation history already contains everything needed to continue once the result exists. Persist that history, and you can discard everything else.

How Infinity works: the slice

Infinity executes each agent as a sequence of bounded slices. In one slice, the runtime loads the agent's history from a pluggable store, runs a single model completion, persists the output, and dispatches at most one tool call. Then the slice ends. A slice never blocks for a tool, a timer, or any other external operation — tool calls are the await points of the system.

Everything that can wake an agent arrives as a message on a shared input queue: user text, tool results, timers, events from other agents. When a message is dequeued, the runtime launches a slice that reloads the agent's state and processes it. Agents live in a pool of threads that share stores, tools and a model; each conversation thread has its own queue, messages for the same thread are processed in order, and slices for different threads run in parallel.

Tools are written differently than in most frameworks: a tool's execute method starts the work and returns immediately, and when the work finishes, the tool sends the result to the agent's input queue itself. The same code runs whether the tool takes a hundred milliseconds or three days.

The 75,000-agent number, read carefully

AWS measured the claim with a benchmark that launched waves of agents in one agent system, backed by a simulated model. Each agent ran twenty turns; each turn made a full round trip through the runtime, with the model streaming a completion of about 1.2 KB that called a tool, the tool result returning through the input queue, and a second completion ending the turn. The run covered 80,000 agents and 3.2 million completions on a single thread of an AMD EPYC 9R14 processor with in-memory stores.

Resident memory grew linearly at about 100 KB per idle agent. At that rate, about 75,000 agents fit in 8 GB — the memory of a Raspberry Pi 5. AWS is explicit that most of that memory is conversation history: an idle agent has no task, no stack and no connection attached to it.

Three caveats belong next to that number. First, the model was simulated, so the benchmark measures the runtime's overhead, not a real deployment with a real model API. Second, histories grow with turns: a production agent with a long, tool-heavy history will cost more than a 100 KB benchmark agent. Third, the Raspberry Pi figure is memory arithmetic, not a deployment recommendation — AWS's own reference deployment runs on cloud services, not on a Pi. The honest headline is "waiting agents cost roughly their history, not a process," rather than "run your agent fleet on a Pi."

Running on serverless: Lambda, SQS, Aurora DSQL

Because a slice never waits and loads all of its state from storage, Infinity can run on serverless platforms, where a function is invoked for a unit of work and may exit as soon as it finishes. Infinity supports this through step mode: the platform calls step once for each batch of delivered messages, and nothing is retained in memory between calls.

AWS's reference deployment uses an SQS FIFO queue as the input queue, with one message group per thread. Each delivery invokes an AWS Lambda function that runs one step and exits. Conversation histories live in Aurora DSQL, deduplication state in DynamoDB, and timers use SQS delivery delays and EventBridge Scheduler. The repository includes CDK constructs that provision these resources.

Five properties make that deployment work. Execution is bounded: a slice is one completion plus a small number of storage and dispatch calls. There is no process affinity: all state is stored by thread ID, so any invocation can process the next message for any thread. Delivery is at-least-once, with stable message IDs and a record of processed IDs per thread, so redeliveries are recognised and skipped. Turns are durable: streamed output is committed only when the turn completes, and a tool call is persisted before it is dispatched. And serialisation needs no distributed locks, because FIFO message groups serialise each thread's slices at the queue. The payoff, in AWS's own example: an agent waiting on a three-day CI pipeline uses no compute until the pipeline reports its result.

Synthetic tool calls: how a waiting agent hears news

Agents that wait also need to learn about events that happen while they wait — a long command producing output over time, or a repository subscription firing each time a pull request opens. Model APIs have no native way to represent an asynchronous event: a conversation is user turns, assistant turns, and tool-call/tool-result pairs, and each tool call must be answered by exactly one result.

AWS considered the obvious workarounds and rejected each for a stated reason. Appending a second result to the original call violates the one-result rule, and models typically reject such conversations. Adding the event as a user message is accepted by the API, but it attributes the event to the user, so the model can no longer distinguish something the user said from something a tool reported. Putting the event in the system prompt removes it from the order of events in the history, which the model relies on to understand when things happened.

Infinity's answer is the synthetic tool call. When an event arrives, the runtime inserts an assistant tool call the model did not actually make — under the dedicated name receive_event__injected — followed by a tool result containing the event. The arguments identify the earlier tool call the event belongs to. The __injected suffix marks the call as runtime-created; the name is never offered to the model as a tool, and the runtime rejects it if the model tries to invoke it.

Parent and child agents without a message bus

The same mechanism carries agent-to-agent communication, so Infinity needs no separate messaging system. Each agent has a built-in spawn_thread tool that creates a child thread with its own input queue. To the parent, spawn_thread behaves like a subscription: the call returns as soon as the child exists, the parent continues or goes idle, and the child's reports arrive later as events on that call — via report_to_parent while it works, and a final report when it calls close_thread. Because each report refers to the original spawn, a parent running many children can tell which child sent what.

There is a subtle choice in where events get processed. By default, each event is handled in a separate child thread that inherits the parent's context: the child can act and report back, or close silently if the event is irrelevant, leaving the parent's history untouched. A tool can instead mark an event as associative, in which case the synthetic call goes directly into the subscribing thread's history — the right choice for events that update an operation the agent is already following, like the output of a test run it started.

What it costs: free code, paid infrastructure

Infinity itself costs nothing: Apache 2.0, no licence fee, no per-agent charge. The bill comes from everything around it, and this is where the fine print matters. First, model tokens. Infinity is a runtime, not a model; every completion is billed by whichever model provider you wire in. The announcement's own code example wires in Anthropic's Claude Sonnet 4.6, but that is an example, not a requirement — you bring your own provider and your own rate card.

Second, AWS services, if you use the reference deployment. Lambda's published pricing charges $0.20 per million requests beyond a free tier of one million requests and 400,000 GB-seconds per month, and SQS, Aurora DSQL and DynamoDB each bill separately on their own meters. A fleet that wakes rarely will sit mostly inside free tiers; a fleet that wakes constantly will not. Either way, the saving is the compute you no longer pay for while agents wait — not the compute, storage and model tokens you still use when they work.

Third, engineering time. This is an early research release, not a managed service with an SLA, a support contract, or a console. You operate the queues, the stores and the deployments yourself — and AWS's own repository had only a small public footprint at announcement time.

Pros and cons

Pros: waiting agents cost roughly their stored history, not a running process; the same agent and tool code runs locally or on serverless; events and multi-agent messages travel through one queue mechanism instead of a separate bus; idempotent, lock-free slice processing fits serverless platforms cleanly; Apache 2.0 with no per-agent fee.

Cons: early research release with no managed offering or SLA; the headline benchmark used a simulated model and in-memory stores, not production conditions; histories — and therefore idle costs — grow with turns; developers must write tools in a non-blocking style, which is a real rewrite for existing harnesses; AWS itself lists open questions on event reasoning, scheduling, compaction and event authenticity.

How Infinity compares

FAQ

Is AWS Infinity a managed service?

No. It is an open-source Rust library and research project. You deploy and operate it yourself, on your own infrastructure or on AWS services you provision. There is no console, no per-agent price and no SLA in the announcement.

Can I really run 75,000 agents on a Raspberry Pi?

Not as a production recommendation. The figure is AWS's benchmark arithmetic: about 100 KB of resident memory per idle agent, measured with a simulated model and in-memory stores, which puts roughly 75,000 agents inside 8 GB. Real agents with long histories and real model APIs will cost more, and AWS's own reference deployment is cloud-based.

What does Infinity cost to run?

The runtime is free under Apache 2.0. You pay for model API tokens from your chosen provider, plus whatever infrastructure you deploy on — in the AWS reference deployment, Lambda, SQS, Aurora DSQL and DynamoDB, each on its own meter. The saving is the compute you stop paying for while agents wait.

What is a synthetic tool call?

It is how Infinity delivers asynchronous events inside the conversation format model APIs already support. The runtime inserts a tool call the model did not make, under the name receive_event__injected, followed by a tool result carrying the event. The name is never offered to the model as a tool, so the model cannot invoke it itself.

Key takeaways

• AWS and UC Berkeley's Infinity suspends agents at every tool call, so a waiting agent costs only its stored conversation history — not a process, a thread, or running compute.
• The 75,000-agents-on-a-Pi figure is AWS's own benchmark with a simulated model: about 100 KB per idle agent, measured with in-memory stores, not a production deployment claim.
• Synthetic tool calls let waiting agents receive events — and parent agents hear from children — inside the conversation format model APIs already understand.
• The AWS reference deployment runs on Lambda, SQS FIFO queues, Aurora DSQL and DynamoDB, with CDK constructs included, and slice processing is idempotent and lock-free by design.
• Infinity is free under Apache 2.0, but it is an early research release with no managed service or SLA — you pay for model tokens and infrastructure, and you operate it yourself.

The most interesting thing about Infinity is not the Pi arithmetic. It is the admission, built into the design, that the agent era's real infrastructure problem is waiting: thousands of agents, each idle most of the time, each billed as if it were working. Whether Infinity becomes the standard answer or just a good idea other runtimes borrow, that problem is not going away — and as of October 24, 2026, one fix for it is free, open source, and waiting on GitHub.

Sources: AWS Open Source Blog, "Introducing Infinity: a runtime for extreme agent concurrency" (October 23, 2026); hydro-project/infinity on GitHub (Apache 2.0); AWS Lambda pricing; Dosa AI Tools daily brief, October 24, 2026. Internal reading: Walmart Lets AI Agents Shop for You and Mastercard Agent Pay for Machines.