Mixtral 8x7B

Mixtral 8x7B is Mistral AI’s sparse mixture-of-experts autoregressive language model from France. Instead of running every feed-forward expert for every token, a router selects two of eight expert blocks at each layer. ‘8x7B’ is therefore not eight independent chatbots, and the full model does not activate all expert parameters for each token. This sparse routing is its key distinction from dense Mistral 7B. TT-Metal lists Mixtral 8x7B at TP=8 on a Wormhole QuietBox with batch 32. Tensor parallelism spreads the model computation over eight devices; that hardware factor happens to equal the number of experts, but the two eights describe different things and should not be conflated.