DeepSeek open-sources DeepInfer, its model-serving engine

Chinese AI company DeepSeek has released DeepInfer, the software it says runs its own large-scale production services, as open-source code, according to the project’s GitHub page. The release is free to use under the Apache 2.0 license.

The code lets outside developers run DeepSeek’s models on their own hardware. But the release is not a complete service. DeepSeek says three parts needed to run one are not included.

The facts: DeepInfer is an inference engine, the software that runs a trained model to answer requests. This release handles offline inference, the project page says.

It supports one model, DeepSeek-V4.1-Flash. By default it downloads the DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face.

It runs on two kinds of chips: Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. Builds need Linux and Python 3.12, plus CUDA 13 for Nvidia or CANN 9.2 for Ascend.

The release includes DeepSeek’s own kernels, among them FlashMLA, DeepGEMM and DeepEP. It also includes a method DeepSeek calls DSpark, which drafts tokens ahead and uses confidence estimates to schedule checks on them.

Other features, according to the page:

• Parallel processing across chips for attention and mixture-of-experts layers.
• Structured outputs that hold a model’s answer to a regex, JSON schema or grammar.
• Batch-invariant inference, which keeps results the same regardless of what other requests share a batch.
• Deterministic sampling, which gives the same output for the same input and settings.

The last two work only on Nvidia hardware, the page says.

DeepSeek lists what is missing for a full service: a KV cache system; separation of the prefill and decode stages, which it says is mostly built; and load balancing across replicas.

DeepSeek says DeepInfer was “originally built primarily on vLLM,” an open-source engine, and drew ideas from SGLang and Nvidia’s TensorRT-LLM.

What it means: DeepSeek has released model weights before. This release covers more of the software it says it uses to serve those models.

Support for Huawei’s Ascend 950 sits alongside Nvidia in the first release. That gives developers using Chinese-made chips an official path to run the model.

The release covers one model and offline use. Teams that want to run a public service would have to build the missing caching and load-balancing parts themselves. The project page posts benchmark charts, but those are DeepSeek’s own tests and have not been independently checked.

What’s next: The project page does not give a timeline for adding the missing parts or more models. Developers can build it from source now. DeepSeek points to its evaluation guide and benchmark scripts for anyone who wants to check results.

Sources: DeepSeek, DeepInfer on GitHub