Identity Matrix Outperforms DeepSeek's mHC in New Qwen3 Training Experiments

Tencent Hy ·

Core Information

Tencent Hunyuan's Hy4 preview highlighted a Zhihu post that questions DeepSeek's mHC (matrix-based head compression), suggesting an Identity matrix may work better. Experiments training Qwen3 1.7B and 8B dense models from scratch on 150B tokens found Identity HC > mHC > mHC-lite > orthogonal mHC.

Key Points

  • The cited Zhihu post, "Your DeepSeek mHC May Not Need the 'm'," argues the doubly stochastic matrix in mHC may be overcomplicated.
  • Training results across Qwen3 1.7B and 8B models show Identity-based head compression outperforms mHC variants.
  • This challenges the default assumption that a learned matrix is necessary for effective head compression in large models.
Loading...