concept-encoder-perceiver_mlm_H512L6C128_20260208_211633

perceiver_mlm_H512L6C128_20260208_211633

Part of the MrCogito research project — Concept Encoder and Decoder: a transformer architecture that compresses long token sequences into a small number of semantic "concept tokens" via cross-attention, then reconstructs or classifies from that compressed bottleneck.

Project page: https://ai.ksopyla.com/projects/concept-encoder/

Architecture

Input tokens [B, L, D_tok]
      │
      ▼  cross-attention (L × layers)
Concept representations [B, C=128, H=512]   ← bottleneck
      │
      ▼  Perceiver IO decoder (position queries → concepts)
Output tokens [B, L, vocab]

Tokens attend to concepts via standard cross-attention. Each encoder layer refines the 128 concept vectors.

Decoder uses input+position queries.

Pretraining

GLUE Fine-tuning

Evaluation Results

Concept-relevant tasks (primary evaluation signal):

Known Limitations

• Concept collapse: Without explicit regularization, the pure MLM objective can collapse concept representations into a low-rank space (effective rank ~5/128). See experiment log.
• CoLA ceiling: Grammatical acceptability requires sub-word patterns that do not survive 4:1 token→concept compression; MCC ≈ 0 is architectural, not a bug.
• GLUE concatenated pairs: Pair tasks (MRPC, QQP, MNLI) encode both sentences into one shared concept set, which compresses the cross-sentence signal.

Usage

import torch
from transformers import AutoTokenizer
import sys
sys.path.append("path/to/MrCogito")  # project root

from nn.concept_encoder import ConceptEncoderConfig
from nn.concept_encoder_perceiver import ConceptEncoderForMaskedLMPerceiver

tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = ConceptEncoderForMaskedLMPerceiver.from_pretrained("ksopyla/concept-encoder-perceiver_mlm_H512L6C128_20260208_211633")
model.eval()

text = "Concept encoders compress tokens into semantic concept vectors."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=512)

with torch.no_grad():
    # Encode: tokens → concept representations [B, C=128, H=512]
    concept_repr = model.encoder(**inputs).last_hidden_state
    # Pool concepts to sentence embedding [B, H=512]
    sentence_embedding = concept_repr.mean(dim=1)

print(sentence_embedding.shape)  # torch.Size([1, 512])

Citation

@misc{mrcogito-concept-encoder-perceiver_mlm_H512L6C128_20260208_211633,
  author       = {Sopyla, Krzysztof},
  title        = {MrCogito Concept Encoder: perceiver_mlm_H512L6C128_20260208_211633},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/ksopyla/concept-encoder-perceiver_mlm_H512L6C128_20260208_211633},
  note         = {Concept bottleneck encoder trained with Masked Language Modeling (MLM)
                  on JeanKaddour/minipile}
}

License

Apache 2.0