DeepSeek Open-Sources DeepInfer: Its Production Inference Engine, Explained

DeepSeek keeps giving away the things other AI labs treat as trade secrets. On October 20, 2026, the Hangzhou lab published DeepInfer — the inference engine it says powers its own large-scale production services — on GitHub under the Apache 2.0 license. Not a research paper. Working code.

What happened

DeepSeek announced DeepInfer on X on Tuesday, October 20, with the line "Open models. Open kernels. Open inference." The repository, deepseek-ai/DeepInfer, is Apache 2.0 licensed, which means commercial self-hosting is explicitly allowed. A companion repository, deepseek-recipe, shipped alongside it — an MIT-licensed token encoder and decoder that, according to reporting on the release, matches how DeepSeek's official API converts text to tokens and back.

The community showed up fast. The repo was created on October 16, its first public commit landed on the 20th, and it had already passed 4,000 stars on day one, climbing past 5,500 stars and 415 forks within a day, per AI Weekly's tracking. For a piece of server infrastructure — not a model, not a demo — that's an unusually loud reception.

It's also a reversal. One detailed breakdown of the release notes that in April 2025 the same team wrote it had decided not to open-source its inference engine. That position has now flipped.

What an inference engine actually does

A trained model, on its own, is just a pile of numbers. An inference engine is the software that turns those numbers into answers people can use: it spreads the model across chips, batches the requests of hundreds of users at once, reuses calculations that don't need repeating, and decides how fast each word comes back.

This layer is where the real money is. The cost per token a company can afford to charge on its API depends heavily on how efficient its serving stack is. DeepSeek lists V4.1 Flash at $0.30 per million uncached input tokens and $1.20 per million output tokens at peak, and half that off-peak, according to AIStockWire — pricing that only makes sense if the serving layer underneath is extremely cheap. DeepInfer is that layer, or at least the closest thing to it that DeepSeek has ever shared.

What's inside DeepInfer

The README reads like a bundle of DeepSeek's recent systems research made operational:

• Custom kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang code — the low-level math routines DeepSeek has open-sourced over the past year, now shipped as a working stack.
• Parallelism across chips: Attention DP with balanced context parallelism, and mixture-of-experts expert parallelism with load balancing (EPLB) for the MoE layers.
• DSpark speculative decoding: a draft-token system that proposes several tokens ahead and uses confidence estimates to decide when to verify them. The default maximum draft length is 5, and MarkTechPost's reading of the project's Q&A says this cuts verified draft tokens by roughly 30% while barely changing accepted length.
• An asynchronous zero-overhead batch scheduler, plus structured outputs constrained by regex, JSON schema, or EBNF grammar.
• Repeatable answers: batch-invariant inference and deterministic sampling with a fixed seed — though both are Nvidia-only in this release.

The README is upfront about lineage: DeepInfer was built primarily on top of vLLM, the most widely used open-source serving engine, with ideas borrowed from SGLang and TensorRT-LLM. DeepSeek also shipped reproducible sweep scripts for its benchmark charts, which were measured on 4 SM100 GPUs for prefill and 64 for decode.

The hardware story: Nvidia and Huawei

This release officially targets two chip families: Nvidia SM100 GPUs (Blackwell, CUDA 13) and Huawei Ascend 950 NPUs (CANN 9.2). It requires Linux, GCC 13 or later, and Python 3.12. That means data-center hardware, not a laptop.

The Huawei support is the part that has infrastructure people talking. AI Weekly's coverage calls it the first first-party production endorsement of Huawei silicon from a frontier lab, and notes that DeepSeek said just last week it plans to pre-train on Huawei hardware by the end of 2026. This isn't a compatibility footnote — it looks like a strategic hedge.

What's missing — read this before you try to replace vLLM

This is an offline inference release, and DeepSeek is honest about what a complete deployment still needs. Three pieces are explicitly left out:

The first release runs a single model, DeepSeek-V4.1-Flash, and only in offline mode. On first run it downloads the October 17 checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, into the Hugging Face cache. As madrobot's writeup puts it, the pieces that turn a fast engine into a cheap public API still have to be built — or borrowed from vLLM or SGLang. Companies hoping to copy DeepSeek's API pricing from this repo alone will be disappointed.

The production numbers, read skeptically

DeepSeek's deployment notes in the repo put a number on the service behind the release: a single cluster serves 155 trillion tokens per day, with a 96% input cache hit rate, roughly 400 milliseconds to first token, and around 230 tokens per second (up to 300 for coding). At off-peak pricing, the notes claim about $38 of revenue per GPU-hour. The plan going forward includes ordinary 8-GPU machines for prefill, 64–128-GPU supernodes for decode, and retaining KV cache for more than 72 hours so repeated context can skip prefill.

All of those figures are DeepSeek's own and have not been independently measured. Treat them as claims, not audits. But even if you discount them heavily, the gap between DeepInfer's open stack and what OpenAI or Anthropic share about their serving layers is the real story here.

Why the timing matters

The release landed one day after the US Department of Homeland Security order made headlines: published in the Federal Register on October 20, it bars DeepSeek products from federal civilian agency purchases and requires agencies to remove DeepSeek products — including third-party products and services built on DeepSeek implementations — from their IT systems. (The signed copy carries an original release date of July 24, per AIStockWire's reporting.)

DeepSeek's stated motive for the release, in its X announcement: "to help the community lower compute costs, reduce energy use, and make AI more affordable." Read generously, that's open-source idealism. Read cynically, it's a lab making sure its infrastructure story is told on its own terms while regulators are circling. Either way, the community keeps the code.

Pros and cons

Pros: real production code under Apache 2.0 (commercial use allowed); dual-chip support including Huawei; reproducible benchmark scripts; structured outputs and deterministic mode for serious work; a companion token tool that matches the official API's behavior.

Cons: one model only; offline inference, not a managed service; three critical serving pieces deliberately omitted; data-center hardware required (Nvidia SM100 or Ascend 950); deterministic mode and batch-invariant inference are Nvidia-only; all performance figures are vendor-reported.

Who should actually try this

If you already serve open-weight models at scale with vLLM or SGLang, DeepInfer is worth a serious look — especially if you run or are evaluating Ascend 950 hardware. If you're a researcher who has ever wondered how DeepSeek serves a $0.30-per-million-token API profitably, the answer is now in your terminal. For everyone else — solo builders, teams without GPU fleets — the managed V4.1 Flash API remains the practical path, and the economics work because of the very engine that was just opened.

FAQ

What is DeepInfer?

DeepInfer is DeepSeek's in-house inference framework — the software that serves its trained models to users. The company open-sourced it on GitHub on October 20, 2026, under the Apache 2.0 license.

Which models and hardware does DeepInfer support?

Only DeepSeek-V4.1-Flash, in offline inference mode. Hardware is limited to Nvidia SM100 GPUs (CUDA 13) and Huawei Ascend 950 NPUs (CANN 9.2), on Linux with GCC 13+ and Python 3.12.

Is DeepInfer free for commercial use?

Yes. Apache 2.0 permits commercial use, and the companion deepseek-recipe tokenizer is MIT-licensed. The main caveat is hardware, not licensing.

Can I use DeepInfer instead of the DeepSeek API?

Not directly. This is offline inference — DeepSeek left out the KV cache system, prefill/decode disaggregation, and replica load balancing that a public API service needs. You'd have to build those yourself or borrow them from vLLM or SGLang.

How does V4.1 Flash's pricing compare?

AIStockWire reports DeepSeek charges $0.30 and $1.20 per million uncached input and output tokens at peak, half that off-peak. For reference, Anthropic lists Claude Haiku 5.5 at $0.10 and $0.50 per million tokens — cheaper on paper, though the models differ in size and capability. Our Claude Haiku 5.5 pricing analysis breaks down the fine print there.

Key takeaways

• DeepInfer is DeepSeek's real production serving engine, open-sourced on October 20, 2026, under the Apache 2.0 license.
• It runs only DeepSeek-V4.1-Flash, offline, on Nvidia SM100 GPUs or Huawei Ascend 950 NPUs.
• It ships DeepSeek's custom kernels, speculative decoding, and async scheduling — built on top of vLLM.
• Three serving pieces are deliberately missing: a KV cache system, prefill/decode disaggregation, and load balancing.
• The timing — a day after the DHS order barring DeepSeek from federal civilian agencies — is as notable as the code, and all performance claims remain vendor-reported.

Published October 21, 2026. Performance figures, repo statistics, and release dates are as reported on October 20–21, 2026; all benchmark and production numbers are DeepSeek's own unless noted.