DeepInfer: DeepSeek's open-source inference framework
DeepInfer is DeepSeek's inference framework for large-scale model serving. The public release supports offline inference with DeepSeek V4.1 Flash; a complete online service needs additional infrastructure.
On this page
What DeepInfer does
DeepInfer runs DeepSeek models on accelerator hardware. DeepSeek says the framework powers its large-scale production inference service. The public repository exposes an offline inference path and core engine features; it is not a ready-made public API service.
What the public release includes
- DeepSeek V4.1 Flash support, optimized kernels, model runtime and sampler runtime.
- Attention and MoE parallelism, asynchronous scheduling, speculative decoding and bounded replay.
- Batch-invariant inference, deterministic sampling and structured outputs, subject to the documented settings and platform limits.
What a complete service still needs
The README explicitly lists a KV cache system, Vision Encoder/Prefill/Decode disaggregation and load balancing across data-parallel replicas as additional work. Do not treat an offline example as proof that a production endpoint is ready.
Before you build
The documented starting point is Linux, Python 3.12 and GCC 13 or later. The current hardware list is NVIDIA SM100 GPUs with CUDA 13 or Huawei Ascend 950 NPUs with CANN 9.2. The example runs DeepSeek V4.1 Flash offline; the default checkpoint downloads on first use unless you provide a local model directory.
DeepInfer and DeepSeek Harness solve different problems
DeepInfer is the model inference layer. DeepSeek Harness (DSH) is an agent environment for sessions, tools and workspaces. This site has not verified a direct DeepInfer integration in DSH. If you plan to connect a self-hosted service, first establish a compatible API endpoint and test it through DSH's provider configuration.
Official sources
Source snapshot checked October 23, 2026. Hardware support and the public release scope may change; check the repository before deployment.