bioRxiv Preprint · 2026

MoE-Bind: Guiding De Novo Protein Binder Generation with Sparse Experts

Dipayan Sarkar¹ · Chiranjib Sarkar^1, ✉

¹ Computational Systems Biology Laboratory, Department of Bioinformatics, University of North Bengal, India

bioRxiv 2026.06.13.732043 · Posted June 13, 2026 · CC-BY 4.0

Paper (PDF) bioRxiv / DOI Video Code BibTeX

MoE-Bind architecture: target sequence tokenized and fed to an MLA-MoE autoregressive decoder that generates a binder sequence. — **Overview of MoE-Bind.** **(a)** The sequence-conditioned autoregressive binder generation pipeline: a target protein sequence is tokenized and decoded into a candidate binder residue by residue, with **no structural input**. **(b)** The MoE-Bind transformer block, pairing RMSNorm, Multi-head Latent Attention (MLA), and a Mixture-of-Experts (MoE) feed-forward layer with skip connections. **(c)** **Multi-head Latent Attention**, which compresses the cached key-value state into a low-rank latent space with decoupled RoPE. **(d)** The sparse **Mixture-of-Experts** layer, routing every residue through top-2 of 8 experts alongside an always-active shared expert, so capacity is decoupled from per-token compute.

Overview video

Trouble playing? Watch on YouTube.

Abstract

De novo protein binder design has been dominated by structure-based pipelines that require known three-dimensional target conformations and consume substantial compute per design, limiting throughput and accessibility. Sequence-only generative models promise a faster, lighter alternative, yet existing systems remain uniformly dense and frequently reintroduce structural computation at inference. Meanwhile, the language modelling community has transitioned from dense designs to sparse Mixture-of-Experts architectures that decouple capacity from per-token compute, a shift that has yet to reach sequence-only protein binder generation. We present MoE-Bind, an autoregressive protein binder generator that, for the first time in this domain, combines Multi-head Latent Attention with a sparse Mixture-of-Experts feed-forward network, evaluated under two independent structure predictors, Boltz-2 and AlphaFold2-Multimer. Despite activating less than half the per-token parameters of compute-matched dense baselines, MoE-Bind matches or exceeds them on full-length receptor-conditioned binder generation on a leakage-free Docking Benchmark 5.0 evaluation, transfers without peptide-specific training to short-peptide design, and reduces training and inference compute by a large margin. Routing analysis reveals interpretable expert specialization at both the individual amino-acid and biochemical-group level, a structured expert-token alignment not previously reported for natural-language MoE models. These results show that sparse architectural design, rather than scale, can deliver fast, structure-free, and interpretable protein binder generation.

100M

total parameters
(~39M active / token)

<50%

per-token params vs.
dense baselines

100s

binders / target
in seconds, 1 GPU

2×

structure predictors
Boltz-2 & AF2-Multimer

Highlights

First target-conditioned protein binder generator to combine Multi-head Latent Attention with sparse Mixture-of-Experts routing.
Matches or exceeds compute-matched dense MHA and GQA baselines while activating under half the per-token parameters.
Generation conditions on the receptor sequence alone; structure is used only as an external check, never inside the design loop.
Experts specialise by individual amino acid and biochemical group (charged, aromatic, aliphatic) with no explicit supervision.
Transfers to short-peptide design, approaching a peptide-specialised reference (PepMLM) without any peptide-specific training.
Evaluated on a leakage-free DB5 benchmark filtered against both the pre-training and fine-tuning corpora.

Interpretable expert specialization

Because proteins have an effective vocabulary of only twenty amino acids that fall into a few well-characterised biochemical classes, MoE-Bind offers an unusually clean window into what its experts learn. Aggregating the router's decisions over all 12 MoE layers and the 22 DB5 complexes reveals that experts organise the amino-acid alphabet along the same biochemical lines biology already recognises, with no explicit supervision toward biochemical grouping. Individual residues show sharp, consistent expert preferences (e.g. the negatively charged residues Asp and Glu both route mainly to Expert 3, and histidine to Expert 0), and these letter-level patterns aggregate cleanly into group-level specialisation.

Three-panel figure showing per-amino-acid expert routing, expert usage by biochemical group, and first-choice enrichment heatmap for MoE-Bind. — **Experts specialise by amino acid and biochemical group.** **(a)** First-choice (top-1) routing per individual amino acid, each bar summing to one over the eight experts. **(b)** Expert usage across the five biochemical groups (combined first- and second-choice routing; raw token counts annotated). **(c)** First-choice enrichment relative to uniform routing (1.0×; red = over-represented, blue = under-represented). **Expert 3** emerges as a charged-residue expert, enriched **2.24×** for negatively charged residues (the strongest value in the matrix), while **Expert 7** becomes a hydrophobic expert, most enriched for both aromatic (1.67×) and nonpolar-aliphatic (1.38×) residues. The two operate on complementary biochemical subspaces, a degree of interpretable specialisation not previously reported for natural-language MoE models.

BibTeX

@article{sarkar2026moebind,
  title   = {MoE-Bind: Guiding De Novo Protein Binder Generation with Sparse Experts},
  author  = {Sarkar, Dipayan and Sarkar, Chiranjib},
  journal = {bioRxiv},
  year    = {2026},
  pages   = {2026.06.13.732043},
  doi     = {10.64898/2026.06.13.732043},
  publisher = {Cold Spring Harbor Laboratory}
}