supersymmetry-v1.1

SuSy Knowledge Base v1.1

A comprehensive, AI-rendered knowledge base of the Supersymmetry (SuSy) modpack for Minecraft 1.12.2 GregTech CEu. Contains 97,289 natural-language documents across 27 types — spanning materials, recipes, machines, items, fluids, ore deposits, process chains, multiblock structures, quest data, cross-modpack material mappings, Minecraft/Forge/GroovyScript internals, and pack architecture. Designed for RAG pipelines, semantic search, and LLM fine-tuning.

This is the v1.1 release — an expansion from the v1 baseline of 81,448 docs (10 types) to 97,289 docs (27 types) via Tracks A–E, P0.1, P0.2, P2.1, P3.1, and P3.2.

Dataset Structure

Each line in corpus.jsonl is a JSON object:

Document Types

Statistics

• Total tokens: ~33.6M (23.5M prompt + 10.1M completion)
• Median doc length: 43 words
• Token distribution (word count, approximate): P10=31, P25=36, P50=43, P75=54, P90=64, P95=71, P99=107
• File size: 38.2 MB (JSONL)
• Source models: DeepSeek V4 Flash (93,877 docs), DeepSeek V4 Pro with thinking (3,412 material docs)

Generation Pipeline

Built from entity_registry.db (42 MB SQLite, 30 tables) — the definitive Supersymmetry data registry — plus BetterQuesting quest files, GroovyScript source, Forge 1.12.x documentation, Minecraft Wiki, and decompiled GTCEu/GCYM/SuSy Java source.

Pipeline:

1. Corpus generation — 25 generator functions in generate_corpus.py, each querying entity_registry.db or parsed source data
2. AI rendering — deepseek_inference.py with dynamic concurrency (4→256), zero 429s, ~$7.65 total inference cost
3. Embedding — hf_embedder.py via BGE-base-en-v1.5 (768-dim) through HuggingFace InferenceClient
4. Indexing — index_builder.py into ChromaDB persistent index (cosine distance)
5. RAG — rag_query_v3.py hybrid BM25 + BGE semantic retrieval → DeepSeek V4 Flash generation

All content is strictly Minecraft 1.12.2. No post-Flattening (1.13+) content is included.

Test Results

20 curated queries, 100% answer relevance, 80% top-result type match.

License

CC-BY-4.0 — free for research and commercial use with attribution.

Known Gaps

1. No vanilla Minecraft recipes — Corpus is GregTech/SuSy-focused. Vanilla crafting/furnace recipes absent.
2. Machine descriptions are formulaic — Machine docs are DB dumps, not functional guides.
3. Late-game sparsity — ZPM+ tiers have proportionally fewer docs (structural, not a bug).
4. No cross-encoder reranker — v3 drops v2's reranker for simplicity. Retrieval quality remains excellent.
5. Fluid material association — Some fluids lack material_resource linkage.

Usage

import json

with open("corpus.jsonl") as f:
    for line in f:
        doc = json.loads(line)
        print(f"[{doc['type']}] {doc['id']}: {doc['text'][:200]}...")
        break

Source

• Database: entity_registry.db (not included — SuSy-specific extraction)
• Generator: generate_corpus.py
• Inference: deepseek_inference.py (DeepSeek API)
• RAG: rag_query_v3.py + ChromaDB + HuggingFace BGE-base