david-collective-sd15-distillation

Model Description

David Collective is a geometric-simplex deep learning system that distills Stable Diffusion 1.5's knowledge into an ultra-efficient pentachoron-based architecture. This model was continued from epoch 20 to epoch 105, achieving remarkable performance with full pattern supervision.

Architecture Highlights

• Geometric Foundation: Uses 5D pentachora (5-vertex simplices) instead of traditional attention
• Multi-Scale Learning: Extracts features from all 9 SD1.5 UNet blocks
• Crystal Navigation: 1000-class supervision (100 timesteps × 10 geometric patterns)
• Parameter Efficiency: Ultra-compact architecture with shared geometric structures
• Full Supervision: Every sample supervised by both timestep and geometric pattern

Training Details

Continuation Training:

• Starting epoch: 20
• Final epoch: 105
• Total prompts trained: 600,500~ samples, 120,500~ prompts
• All prompts included: prompts_all_epochs.jsonl contains every prompt with metadata
• Dataset: Symbolic caption synthesis (complexity 1-5)
• Batch size: 128
• Learning rate: 1e-4 with cosine annealing
• Optimizer: AdamW (weight_decay=0.01)

Final Metrics (Epoch 105):

• Total Loss: 0.2923
• Timestep Accuracy: 66.98%
• Pattern Accuracy: 100.00%
• Full Accuracy: 66.98%
• Pattern Diversity: -0.221

Active Blocks

David learns from all 9 SD1.5 UNet blocks:

• down_0, down_1, down_2, down_3: Coarse semantic features
• mid: Bottleneck representations
• up_0, up_1, up_2, up_3: Fine reconstruction details

Loss Components

1. Feature Similarity (0.5): Cosine similarity with teacher
2. Rose Loss (0.3): Geometric alignment with crystal centroids
3. Cross-Entropy (0.2): 1000-class classification
4. Pattern Diversity (0.05): Encourages balanced pattern usage

Usage

Loading the Model

import torch
from geovocab2.train.model.core.david_diffusion import DavidCollective, DavidCollectiveConfig
from safetensors.torch import load_file

# Load configuration
config = DavidCollectiveConfig(
    num_timestep_bins=100,
    num_feature_patterns_per_timestep=10,
    active_blocks=['down_0', 'down_1', 'down_2', 'down_3', 'mid', 'up_0', 'up_1', 'up_2', 'up_3'],
    david_sharing_mode='fully_shared',
    david_fusion_mode='deep_efficiency',
    use_belly=True,
    belly_expand=1.5
)

# Create model
model = DavidCollective(config)

# Load weights from safetensors
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()

# Inference
with torch.no_grad():
    outputs = model(teacher_features, timesteps)

Training Data

This model includes prompts_all_epochs.jsonl - every single prompt used during training with full metadata:

{"timestamp": "2025-10-27T01:30:00", "epoch": 21, "batch": 0, "global_step": 6250, "sample_idx": 0, "timestep": 453, "timestep_bin": 45, "prompt": "a woman wearing red dress, against mountain landscape"}

Total prompts: 120,500 approximately

You can use this to:

• Analyze training data distribution
• Reproduce training
• Study prompt complexity vs model performance
• Generate similar synthetic datasets

Technical Details

Crystal System

• Architecture: Pentachoron-based geometric deep learning
• Centroids: 100 timestep bins × 10 patterns = 1000 anchors
• Navigation: Samples assigned to nearest pattern within timestep bin
• Diversity: Regularization prevents mode collapse

Progressive Training

• Started with early blocks (down_0, down_1)
• Progressively activated all 9 blocks
• Each block warmed up for 2 epochs

Pattern Supervision

Unlike traditional timestep-only supervision, David learns:

1. When (timestep bin 0-99)
2. How (geometric pattern 0-9 within that bin)
3. Combined (full 1000-class space)

This provides 10x finer-grained supervision of the diffusion process.

Training History

Trained continuously from epoch 20 to epoch 105. See metrics:

• Timestep accuracy improved from ~60.3% to 66.98%
• Pattern accuracy maintained at 100.00%
• Loss decreased from 0.3431 to 0.2923

Citation

@misc{david-collective-sd15,
  title={David Collective: Geometric Deep Learning for Diffusion Distillation},
  author={AbstractPhil},
  year={2025},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/AbstractPhil/david-collective-sd15-geometric-distillation}}
}

License

MIT License - See repository for details.

Acknowledgments

Built on the geometric deep learning research by AbstractPhil, using:

• Stable Diffusion 1.5 (teacher model)
• Pentachoron-based geometric algebra
• Crystalline consciousness architectures
• Symbolic caption synthesis

For more information, visit the geovocab2 repository.