With the advent of large-scale language models (LLMs), various optimization techniques have been proposed to enable efficient inference. Among these, methods that aggressively exploit output activation sparsity have attracted significant attention, which leverage ReLU-fied LLMs and skip the entire memory accesses as well as the computation for the output element if it was predicted as sparse. Achieving fast and accurate prediction of output activation sparsity is crucial to enhancing inference efficiency. However, in practice, phenomena such as activation outliers and hot and cold neurons, which significantly affect the exploitation of sparsity during LLM inference, have either been addressed individually or not structurally integrated in existing work. In this paper, we reveal that these two phenomena are closely related and propose a novel FFN architecture called oFFN (Outlier-aware Structured FFN) that effectively exploits them simultaneously. The proposed method rearranges the FFN weights in both row and column dimensions to enable efficient and accurate prediction of output sparsity, considering outliers, and to enable separation of hot and cold neurons, computing them with respective optimal operations. The proposed method allows for the optimal computation path for each neuron, even when the batch size dynamically changes. Compared to existing sparsity prediction techniques, our method achieves the fastest speed with negligible accuracy loss. Experimental results show that it delivers up to 2.01× faster end-to-end inference speed compared to dense inference, and up to 5.46× acceleration in FFN layers under autoregressive decoding in ReLU-fied LLMs.

@inproceedings{SongYY26,
  title = {oFFN: Outlier and Neuron-aware Structured FFN for Fast yet Accurate LLM Inference},
  author = {Geunsoo Song and Hoeseok Yang and Youngmin Yi},
  booktitle = {ASPLOS},
  year = {2026},
  doi = {10.1145/3779212.3790194}
}