Downloads · 30 days
3
0% of all-time downloads
parthvpatil18/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab is a feature extraction model from parthvpatil18. Use it when you need embeddings to search or compare text. The card lists the license as mit.
ModelAutoEncoder is a specialized autoencoder designed to compress high-dimensional pre-trained model parameter vectors (8192-dim) into compact 512-dimensional embeddings that preserve essential architectural characte…
Downloads · 30 days
3
0% of all-time downloads
All-time downloads
3.6K
Public
Repo size
214 MB
Likes
1
Public
Click a slice to open those files.
.bin214 MB · 100%
From the Hugging Face model README
ModelAutoEncoder is a specialized autoencoder designed to compress high-dimensional pre-trained model parameter vectors (8192-dim) into compact 512-dimensional embeddings that preserve essential architectural characteristics. This model is part of a research project on pre-trained model recommendation for specific hardware using transformers.
This autoencoder enables:
Input [batch, 8192]
↓ (z-score normalization)
↓ Linear(8192 → 1024) + bias
↓ GELU activation
↓ Dropout(p=0.05)
↓ Linear(1024 → 512) + bias ← Bottleneck (encoded output)
↓ Linear(512 → 1024) + bias
↓ GELU activation
↓ Dropout(p=0.05)
↓ Linear(1024 → 8192) + bias ← Reconstruction output
| Metric | Value |
|---|---|
| Best Validation Loss | 0.246083 |
| Best Training Loss | 0.262606 |
| Best Epoch | 4 |
| Training Epochs | 5 (early convergence) |
| Batch Size | 16 |
| Learning Rate | 0.001 → 0.00001 (ReduceLROnPlateau) |
Epoch 1: train_loss=0.439, val_loss=0.432
Epoch 2: train_loss=0.440, val_loss=0.363
Epoch 3: train_loss=0.359, val_loss=0.299
Epoch 4: train_loss=0.300, val_loss=0.246 ← Best
Epoch 5: train_loss=0.263, val_loss=0.253
The model converged rapidly with stable training dynamics, achieving best validation performance at epoch 4.
| Metric | Value |
|---|---|
| Model Size (FP32) | 68.04 MB |
| Model Size (FP16) | 34.02 MB |
| Inference Latency (single) | ~0.1ms on V100 GPU |
| Inference Latency (batch=256) | ~2ms on V100 GPU |
| Throughput | 100,000 samples/second |
| Memory (inference) | ~273 MB (model + optimizer states) |
pip install torch transformers
import torch
from transformers import AutoModel
# Load the model from HuggingFace Hub
model = AutoModel.from_pretrained("parthvpatil18/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab", trust_remote_code=True)
model.eval()
# Move to GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
import torch
import numpy as np
# Example: 8192-dim model parameter vector (normalized)
# IMPORTANT: Apply z-score normalization with training dataset statistics
model_params = torch.randn(1, 8192).to(device) # Replace with your normalized data
# Extract 512-dim embedding
with torch.no_grad():
encoded, reconstructed = model(model_params)
print(f"Input shape: {model_params.shape}") # [1, 8192]
print(f"Embedding shape: {encoded.shape}") # [1, 512]
print(f"Reconstruction shape: {reconstructed.shape}") # [1, 8192]
# Process multiple models at once
batch_params = torch.randn(32, 8192).to(device) # Batch of 32 models
with torch.no_grad():
embeddings, reconstructions = model(batch_params)
# Use embeddings for downstream tasks (similarity search, clustering, etc.)
For a complete example of using this model in the pre-trained model recommendation pipeline, see the GitHub repository.
# Extract model parameters using specialized extractors
from extractors import HuggingFacePipelineExtractor
extractor = HuggingFacePipelineExtractor()
raw_params = extractor.extract("your-model-name") # Returns 8192-dim vector
# Apply normalization (CRITICAL STEP)
# You must use mean/std from the training dataset
normalized_params = (raw_params - train_mean) / train_std
# Generate embedding
with torch.no_grad():
embedding, _ = model(torch.tensor(normalized_params).unsqueeze(0).to(device))
# Use embedding for model recommendation
print(f"Model embedding: {embedding.shape}") # [1, 512]
The model requires z-score normalization of input data using statistics computed from the training dataset:
normalized_input = (raw_input - training_mean) / training_std
Failure to normalize inputs will result in poor performance or invalid embeddings.
[batch_size, 8192]torch.float32The model returns a tuple (encoded, reconstructed):
encoded: Shape [batch_size, 512], dtype torch.float32
reconstructed: Shape [batch_size, 8192], dtype torch.float32
Smooth L1 Loss (Huber Loss) + L1 Regularization:
total_loss = SmoothL1(reconstructed, input, β=0.5) + 0.001 × |encoded|.mean()
The model exhibits stable gradient flow due to:
If you use this model in your research, please cite:
@misc{modelautoencoder2024,
title={ModelAutoEncoder: Pre-trained Model Parameter Embeddings for Recommendation Systems},
author={Parth Patil},
year={2024},
url={https://github.com/Parth1811/ptm-recommendation-with-transformers},
howpublished={HuggingFace Hub: \url{https://huggingface.co/parthvpatil18/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab}}
}
Full source code, training scripts, and evaluation tools are available at:
https://github.com/Parth1811/ptm-recommendation-with-transformers
model/model_encoder.py: AutoEncoder implementationtrain/model_autoencoder_trainer.py: Training pipelinedataloader/model_npz_dataset.py: Dataset loader for model parametersextractors/: Tools for extracting parameter statistics from pre-trained modelsconfig.ini: Full configuration file with hyperparameters# Clone the repository
git clone https://github.com/Parth1811/ptm-recommendation-with-transformers.git
cd ptm-recommendation-with-transformers
# Install dependencies
pip install -r requirements.txt
# Train the autoencoder
python train.py ModelAutoEncoderTrainer
# Extract embeddings for all models in a directory
python evaluate_model_autoencoder.py
This model is part of a research thesis project on pre-trained model recommendation for specific hardware using transformers. The system learns to recommend which pre-trained models will perform best on given datasets by extracting compact embeddings from model parameters and dataset features.
For questions, issues, or collaboration inquiries:
Last updated: November 2024