DeepInfer: DeepSeek open-sources its inference engine, but it runs one model, offline, for now

DeepInfer is the open-source inference engine from DeepSeek, the software that loads a model and turns prompts into text on GPUs. The README calls it the framework behind DeepSeek's production services, and this first release covers offline inference only. It runs one model, DeepSeek-V4.1-Flash, on two kinds of hardware. For most people the practical answer today is that vLLM or SGLang still serve a wider range of models and machines.

Repo

deepseek-ai/DeepInfer

Licence

Apache-2.0

Maintained by

DeepSeek-AI (no individual maintainers named)

Activity

10.3k stars, 900 forks, 80 open pull requests, 3 commits on main (GitHub, 20 Oct 2026)

Models

DeepSeek-V4.1-Flash only (default checkpoint DeepSeek-V4.1-Flash-1017)

Hardware

NVIDIA SM100 GPUs, Huawei Ascend 950 NPUs

What it does

Per the README, DeepInfer was built mainly on vLLM, with ideas from SGLang and TensorRT-LLM, and it uses DeepSeek's own kernels (small GPU programs): FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. Listed features:

• Parallelism options for spreading a model across GPUs: attention data parallelism, balanced context parallelism, and expert parallelism with load balancing for the mixture-of-experts layers.
• DSpark, a draft-token decoding method (the engine guesses several tokens ahead and verifies them), switched on with an environment variable.
• Asynchronous scheduling, on by default.
• Batch-invariant inference and deterministic sampling, so the same prompt gives the same output. These work on NVIDIA only.
• Structured outputs from a regex, a JSON schema or a grammar.

What is missing

• Done: Offline inference with the core serving features
• Done: Structured outputs
• Not done: A KV cache system (the stored attention state that lets a server reuse earlier work)
• Not done: Prefill/decode disaggregation (running the two phases on separate machines)
• Not done: Load balancing across data-parallel replicas
• Not done: Online or API serving (only offline use is documented)

The README says the missing pieces are needed for a complete inference service. It publishes charts for prefill and decode speed, measured on 4 and 64 SM100 GPUs, but the page gives no numbers in text. A sweep script (examples/sweep/run_sweep.sh) is included to reproduce them.

Try it

The prerequisites are heavy: Linux, GCC 13 or later, Python 3.12, and CUDA 13 for NVIDIA or CANN 9.2 for Ascend.

1. Make an environmentCreate an empty Python 3.12 environment and activate it.
2. BuildSet DEEPINFER_ACCELERATOR=nvidia (or ascend) and run ./build.sh.
3. Runpython examples/offline_inference.py. The checkpoint downloads from Hugging Face on first run; set DEEPINFER_HF_MODEL_DIR to use a local copy.

What it replaces

This table uses what each project's README lists as of 20 Oct 2026.

Star counts show attention and say nothing about quality. No independent benchmark compares the three on the same hardware, and DeepInfer's own charts have no numbers in the README. For any model other than V4.1-Flash, or any machine other than SM100 and Ascend 950, DeepInfer is not an option yet.

The $38 per GPU-hour figure

The repo says nothing about revenue. A $38 per card per hour figure, and a claim that one cluster serves 155 trillion tokens a day at off-peak pricing, circulate in a Hacker News comment quoting a Chinese-language post and in a post on X. Nobody in the discussion verified either number. Commenters also pointed out that it would be revenue, which is a different figure from profit.

Who's behind it

DeepSeek-AI publishes the repo under its own organisation, with no individual maintainers listed. The Hacker News post had 1,703 points and 621 comments as of 20 Oct 2026. Much of the discussion is about inference economics and whether AI becomes a commodity, with little on the engine itself.

Best for / skip if

Sources

1. deepseek-ai/DeepInfer: licence, features, hardware, install steps, missing components, stars
2. vllm-project/vllm: licence, stars, serving, hardware and feature list
3. sgl-project/sglang: licence, stars, hardware list
4. Hacker News discussion: the $38 and 155T figures, reader reaction