Tencent Hunyuan Hy4-preview Runs in vLLM from Day One on NVIDIA GPUs

vLLM ·

核心信息

Tencent Hunyuan's Hy4-preview is now supported in vLLM from day one, verified on NVIDIA GPUs, with sparse MoE and sparse attention designed for efficient inference.

要点

  • Architecture: 770B total / 49B active parameters, 256 routed experts plus one shared expert, and 1M context length; each query attends to only 2048 tokens.
  • Sparse index reuse: only 21 of 78 layers compute their own sparse index; the other 57 reuse one, cutting overhead.
  • Built-in MTP: a 10B multi-token prediction layer ships in the checkpoint (0.7B active, draft depth 3).
  • Serving: Tencent's HPC-Ops attention and MoE kernels are already in vLLM main since Hy3; use VLLM_ENABLE_HPC_OPS=1 with vllm serve tencent/Hy4-preview-FP8...
Loading...