Research explores uncertainty estimation in Mixture-of-Experts models

A master's thesis from the University Library Linz explores methods for recovering epistemic uncertainty in large language models (LLMs). The research, authored by Benedikt-Luka Antic under the advisement of Sepp Hochreiter, addresses the high cost of using explicit ensembles for uncertainty estimation in modern, large-scale models.

The study investigates whether alternative computational paths within a single pretrained Sparse Mixture-of-Experts (MoE) model can provide useful uncertainty signals during inference. Using the Mixtral-8x7B-v0.1 model, the proposed method introduces Gumbel noise to the Top-2 expert-selection step during continuation processing. This approach allows diversity to arise from routing rather than token sampling, while keeping pretrained parameters and original router probabilities unchanged.