manifoldgl

ManifoldGL – Information‑Geometric Adapter for LLMs

ManifoldGL is a parameter‑efficient adapter that enforces hyperbolic geometry on the latent space of large language models. It treats the meaning of a token as a fiber over a hyperbolic base manifold (a Poincaré ball), rather than a single vector in flat Euclidean space. Latent states are projected onto the ball, and attentions are computed using geodesic distance. A sheaf‑theoretic consistency loss and natural gradient optimization maintain semantic structure during training.

Motivation and theoretical background

Modern LLMs embed tokens in a Euclidean vector space. While convenient, Euclidean geometry has limited capacity to represent hierarchical structures: flat space grows polynomially, whereas hierarchical trees expand exponentially. By contrast, hyperbolic space grows exponentially and preserves both local and global relationships in a hierarchy【247949143190903†L115-L124】. Hyperbolic embeddings outperform Euclidean ones for lexical entailment, similarity and analogy tasks【247949143190903†L154-L169】. ManifoldGL leverages these properties by modelling the latent space as a fiber bundle over a hyperbolic base: each point in the Poincaré ball encodes a context, and its fiber contains a distribution of semantic components.

Results on ARC‑AGI benchmark

ManifoldGL fine‑tuned on Qwen2.5‑7B improves task accuracy on the ARC‑AGI benchmark from 12.4 % to 28.7 %, a 131.5 % relative improvement. The model also achieves a Manifold Faithfulness Rate (MFR) of 94.2 %, indicating high adherence to the hyperbolic constraints, and maintains a curvature close to the target κ = ‑1 (mean ‑0.98 ± 0.04). Ablation studies show that removing curvature regularization, natural gradients, sheaf consistency or the hyperbolic target significantly reduces accuracy; the Euclidean target ablation causes the largest drop (–10.9 %), highlighting the importance of hyperbolic geometry.

Files in this repository

This model card accompanies adapter weights trained with ManifoldGL. The files follow the structure of the original repository:

• adapter_config.json – configuration for PEFT/LoRA loading
• pytorch_adapter.bin – adapter weights
• README.md – this model card

Quick start

from transformers import AutoModelForCausalLM
from peft import PeftModel

# Load the base model (Qwen2.5-7B)
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B")

# Load the ManifoldGL adapter
model = PeftModel.from_pretrained(base_model, "jesusvilela/manifoldgl")

# Now use model.generate(...) to generate text with hyperbolic adapters

Usage

This adapter can be loaded with PEFT on top of any compatible Qwen2.5‑7B model. During generation, latent states are projected into hyperbolic space and meaning is represented as fibers. We recommend using FP32 precision for maximum stability.

Citation

If you use ManifoldGL in your work, please cite the accompanying thesis and repository.