concept-first-codegen

Concept-First Code Generation

Inspired by VL-JEPA: Predict concept embeddings first, then generate code conditioned on them.

The Idea

Traditional autoregressive models predict tokens one at a time, which can lead to losing coherence or hallucinating APIs. The Concept-First approach solves this by:

1. Concept Encoder: Encoding code snippets into semantic embeddings.
2. Concept Predictor: Predicting what the code embedding should look like given a query.
3. Concept-Conditioned Generation: Retrieving similar concepts to guide the LLM.

graph LR
    A[Query] --> B(Concept Predictor)
    B --> C{Concept Space}
    C --> D[Retrieve Similar Code]
    D --> E[Conditioned Generation]

Models Used (January 2026)

Files in this Repo

• concept_predictor.pt: PyTorch weights for the concept predictor MLP.
• concept_predictor.gguf: GGUF format for edge deployment (llama.cpp/LM Studio).
• concept_bank.pt: Pre-computed embeddings for the concept retrieval bank.

Usage

# Load the concept predictor
import torch
checkpoint = torch.load("concept_predictor.pt")
# ... (See Colab notebook for full implementation)

Datasets

Constructed from high-quality subsets of:

• MBPP
• Evol-Instruct-Code
• Magicoder-OSS-Instruct

Credits

Created by Core Subagent (Colab Composer) for Riley Seaburg.