longformer-secret-masker

Longformer Secret Masker (Long Expert)

Long Expert model for SecMask MoE system - specialized for detecting secrets in long-context documents (up to 2048 tokens).

---

🎯 Overview

Fine-tuned Longformer model for detecting and classifying secrets (API keys, tokens, credentials) in long-form text using Named Entity Recognition (NER). Serves as the Long Expert in the SecMask Mixture of Experts architecture, handling complex cases requiring extended context (7.3% of requests) with ~12ms latency.

⚠️ IMPORTANT: This model has known compatibility issues with the current inference pipeline. Some test cases (PEM blocks, K8s secrets) produce errors. The recommended production configuration is Fast Expert + Filters alone, which achieves better results (92.3% precision, 80% recall) than Full MoE. This Long Expert model is experimental and optional. See Configuration Guide for details.

Key Features

✅ Extended Context: 2048 token limit (4x larger than DistilBERT)
✅ High Precision: 82% (low false positives)
✅ Complex Structures: Handles multi-line YAML, K8s manifests, large configs
⚠️ Experimental: Known compatibility issues, not required for production
✅ Multi-Secret Types: AWS keys, GitHub tokens, JWTs, API keys, PEM blocks, K8s secrets

Note: Combines NER model (82% precision, 38% recall) with deterministic post-filters for comprehensive long-document secret detection. However, due to compatibility issues, Fast Expert + Filters achieves better results in practice.

Ideal For

🎯 Kubernetes Manifests (1000+ line YAML files)
🎯 Terraform/CloudFormation configs
🎯 Docker Compose files
🎯 Large Log Files with embedded secrets
🎯 Multi-file Concatenated outputs
🎯 API Response Payloads (JSON arrays)

---

🚀 Quick Start

Installation

pip install transformers torch

Basic Usage

Standalone (Direct):

from transformers import pipeline

# Load model
classifier = pipeline(
    "token-classification",
    model="AndrewAndrewsen/longformer-secret-masker",
    aggregation_strategy="simple"
)

# Detect secrets in long config file
with open('docker-compose.yml', 'r') as f:
    config = f.read()  # May be >512 tokens

results = classifier(config)

print(results)
# [{'entity_group': 'SECRET', 'score': 0.92, 'word': 'AKIAIOSFODNN7EXAMPLE', ...}, ...]

Recommended (via SecMask MoE):

# Clone SecMask repo
# git clone https://github.com/AndrewAndrewsen/secmask.git

from infer_moe import mask_text_moe

# Router automatically uses Long Expert for complex texts
masked = mask_text_moe(
    long_yaml_content,  # >512 tokens or multi-line structure
    fast_model_dir="AndrewAndrewsen/distilbert-secret-masker",
    long_model_dir="AndrewAndrewsen/longformer-secret-masker",
    tau=0.80,
    routing_mode="heuristic"
)

print(masked)
# Secrets in long document masked

Command Line (via SecMask)

# Clone repo
git clone https://github.com/AndrewAndrewsen/secmask.git
cd secmask

# Mask secrets in long file
python infer_moe.py \
    --in k8s-deployment.yaml \
    --fast-model AndrewAndrewsen/distilbert-secret-masker \
    --long-model AndrewAndrewsen/longformer-secret-masker \
    --routing-mode heuristic \
    --tau 0.80 \
    --out k8s-deployment-masked.yaml

---

📊 Performance

Secret Detection Metrics

⚠️ Note: This model has compatibility issues with the current inference pipeline. Evaluation shows errors on PEM block and K8s secret test cases. The Full MoE configuration (Fast+Long+Filters) achieves 90.9% precision and 76.9% recall due to these errors, compared to 92.3% precision and 80% recall for Fast+Filters alone. See BENCHMARK_RESULTS.md for details.

When This Model Is Used (MoE Routing)

The router would select this Long Expert when:

• Token count > 512
• Contains PEM blocks (-----BEGIN ... -----END ...)
• Contains K8s manifests (kind: Secret)
• Multi-line structured content (YAML, JSON, config files)
• Coverage: 7.3% of real-world requests

⚠️ Recommendation: Due to compatibility issues, use the DistilBERT Secret Masker with post-processing filters instead. This achieves better results without the Long Expert.

---

🏗️ Model Details

Architecture

• Base Model: allenai/longformer-base-4096 (149M params, Apache 2.0)
• Task: Token Classification (NER)
• Max Sequence Length: 2048 tokens (effective limit for inference)
• Attention: Sliding window + global attention
• Label: B-SECRET, I-SECRET, O (BIO tagging)

Training Details

• Dataset: Custom SecretMask v2 (6,000 training examples, filtered for long contexts)
• Optimizer: AdamW (lr=3e-5)
• Epochs: 3
• Batch Size: 4 (with gradient accumulation)
• Hardware: GPU (NVIDIA A100 or equivalent)
• Training Time: ~2 hours

Evaluation

Evaluated on 600 held-out examples from SecretMask v2 test set:

Precision: 0.82
Recall: 0.38
F1: 0.52
Support: 1,021 secret tokens

Key Insights:

• High Precision (82%): Very low false positive rate - production-safe
• Lower Recall (38%): Conservative detection when used standalone
• Long-Context Strength: Excels at multi-line secrets (PEM certificates, large YAML configs) where DistilBERT cannot provide full context
• Production Strategy: Deploy within MoE system with deterministic filters for comprehensive coverage
• Threshold Tuning: Adjust τ parameter for recall/precision trade-off

---

💡 Use Cases

Production Applications

1. Infrastructure-as-Code (IaC) Scanning

   • Terraform .tf files
   • CloudFormation .yaml templates
   • Ansible playbooks
   • Pulumi configs

2. Kubernetes Security

   • Scan manifests for hardcoded secrets
   • Audit kind: Secret resources
   • Pre-deployment validation

3. Configuration Auditing

   • Docker Compose files
   • .env files (concatenated)
   • Large application configs

4. Log File Sanitization

   • Multi-line stack traces with secrets
   • Aggregated logs from multiple services

5. Documentation Cleanup

   • Example configs in docs
   • Tutorial files with placeholder secrets

Example: Kubernetes Manifest Scanning

from transformers import pipeline

classifier = pipeline(
    "token-classification",
    model="AndrewAndrewsen/longformer-secret-masker"
)

# Read K8s manifest
with open('deployment.yaml', 'r') as f:
    manifest = f.read()

# Detect secrets
secrets = classifier(manifest)

if secrets:
    print(f"⚠️  Found {len(secrets)} secrets in manifest!")
    for secret in secrets:
        print(f"  - {secret['word'][:20]}... (score: {secret['score']:.2f})")

Example: Terraform State File Audit

import json
from transformers import pipeline

classifier = pipeline(
    "token-classification",
    model="AndrewAndrewsen/longformer-secret-masker"
)

# Read Terraform state (can be very long)
with open('terraform.tfstate', 'r') as f:
    state = f.read()

# Detect secrets
secrets = classifier(state[:4096])  # Truncate if >4096 tokens

# Flag resources with secrets
if secrets:
    print("❌ Secrets detected in Terraform state!")
    print("Consider using encrypted backends or secret managers.")

See SecMask Examples for more.

---

⚠️ Limitations

Known Issues

1. Token Limit: Cannot handle texts >2048 tokens (requires chunking)
2. Higher Latency: ~12ms vs 6ms for DistilBERT (2x slower)
3. English Only: Trained on English text
4. False Negatives: ~25% recall means some secrets may be missed
5. Memory Usage: Requires ~2GB GPU memory for inference

Not Suitable For

❌ Short texts (<100 tokens) - use distilbert-secret-masker for speed
❌ Real-time streaming (higher latency)
❌ Non-English text
❌ Binary data or encrypted content
❌ Extremely long documents >4096 tokens (use chunking)

Recommended Mitigations

• Use MoE routing: Automatically routes short texts to Fast Expert
• Chunk long documents: Split >2048 token files into segments
• Combine with filters: Use deterministic filters for PEM blocks, K8s secrets
• Adjust threshold: Lower tau for higher recall (more false positives)
• GPU recommended: For production use with multiple files

---

📜 License & Attribution

Model License

Apache 2.0 (inherited from allenai/longformer-base-4096)

Base Model Attribution

This model is fine-tuned from:

• Model: allenai/longformer-base-4096
• Authors: Allen Institute for AI (AI2)
• License: Apache 2.0
• Citation:

  @inproceedings{beltagy2020longformer,
    title={Longformer: The Long-Document Transformer},
    author={Beltagy, Iz and Peters, Matthew E. and Cohan, Arman},
    booktitle={arXiv:2004.05150},
    year={2020}
  }
  

SecMask Code License

The SecMask inference code and training scripts are licensed under MIT. See GitHub repo.

---

🔗 Related Models

---

📚 Resources

• GitHub Repository: AndrewAndrewsen/secmask
• Documentation: README
• Benchmarks: BENCHMARKS.md
• Examples: EXAMPLES.md
• Use Cases: USE_CASES.md
• Deployment: DEPLOYMENT.md

---

🤝 Contributing

Issues and contributions welcome! See CONTRIBUTING.md.

---

Developed by: Anders Andersson (@AndrewAndrewsen)
Part of: SecMask MoE System