Posts tagged #MoE

• How to Honestly Test if a Neural Network Can Be Compressed

  Apr 28, 2026

  Pre-registration, trap cells, τ-hardened baselines, and kill-fast protocols: a field methodology for compression research that tries to kill its own ideas. With actual results from OLMoE-1B-7B.

  #ml-research #compression #methodology #quantization #MoE

• Part E Pivot: FFN Rotation and the Narrow-d Falsification

  Apr 25, 2026

  After the KV-cache gauge, the obvious next move was applying β-lift to FFN weights. We tested it. It failed. Here is what the RAdam convergence probe and the 1-bit generation test actually showed.

  #ml-research #quantization #MoE #compression #MLP

• Phase-Collapse Defragmentation: Why MoE KV-Cache Resists 1-bit Quantization

  Apr 23, 2026

  Attention head activations in Mixture-of-Experts models cluster around expert routing patterns. Quantizing the KV-cache destroys this signal. The MoEGauge framework builds provable bounds on exactly how much.

  #ml-research #quantization #MoE #compression #KV-cache