Skip to content
LiveNew tools this week
Submit
AIMarketly
llm-code-docs logo

LLM Code Docs

llm-code-docs

Centralized AI-readable documentation repository with 245+ frameworks and tools. Use to find documentation, add new sources, or update existing docs. Located at ~/github/llm-code-docs.

majiayu000/claude-skill-registry0installs662stars

SKILL.md

Full skill instructions

LLM Code Docs

Overview

Centralized repository of AI-optimized documentation for 245+ frameworks, libraries, and developer tools. Three-tier structure prioritizing llms.txt standard sites.

Location: ~/​github/​llm-code-docs

<examples> <example> Task: Find documentation for FastAPI to understand authentication patterns 
grep -i "fastapi" ~/​github/​llm-code-docs/​index.yaml
ls ~/​github/​llm-code-docs/​docs/​github-scraped/​fastapi/
cat ~/​github/​llm-code-docs/​docs/​github-scraped/​fastapi/​docs/​advanced/​security.md

Output: FastAPI documentation available in github-scraped folder with complete API security patterns and examples. </​example>

<example> Task: Search across all documentation for "middleware" implementations 
grep -r "middleware" ~/​github/​llm-code-docs/​docs/​llms-txt/​*/​llms-full.txt | head -10
grep -r "middleware" ~/​github/​llm-code-docs/​docs/​github-scraped/ --include="*.md" | head -10

Output: Multiple implementations found across FastAPI, Express, Django, and other frameworks in the repository. </​example>

<example> Task: Add documentation for a new framework that has an llms.txt file 
# Check if llms.txt exists
./​scripts/​find-llms-txt.sh example.com

# Add to config
echo "  - name: example-framework
    base_url: https://docs.example.com/
    description: Example framework documentation" >> scripts/​llms-sites.yaml

# Download documentation
python3 scripts/​llms-txt-scraper.py --site example-framework

# Update index
python3 scripts/​update-index.py

Output: New framework documentation indexed and available at docs/​llms-txt/​example-framework/ </​example>

<example> Task: Update all documentation sources to get latest versions 
cd ~/​github/​llm-code-docs
./​scripts/​update.sh

Output: All 228+ llms.txt sites, 14 Git repositories, and web-scraped sources refreshed with latest content. </​example>

<example> Task: Find documentation for a specific library and verify it exists 
ls ~/​github/​llm-code-docs/​docs/​llms-txt/ | grep -i playwright
head -20 ~/​github/​llm-code-docs/​docs/​llms-txt/​playwright/​llms-full.txt

Output: Playwright documentation found and verified to contain expected content. </​example> </​examples>

First-Time Setup

Initial Configuration

# Navigate to the repository
cd ~/​github/​llm-code-docs

# Install Python dependencies
pip install -r requirements.txt

# Verify documentation is up-to-date
./​scripts/​update.sh

Verify Installation

# Check available documentation
ls docs/​llms-txt/ | wc -l        # Should show ~228
ls docs/​github-scraped/           # Should show ~14 repos
ls docs/​web-scraped/              # Should show custom scrapers

# Verify index
cat index.yaml | head -20

Repository Structure

llm-code-docs/
├── docs/
│   ├── llms-txt/           # 228 sites (HIGHEST PRIORITY)
│   ├── github-scraped/     # 14 Git repo extractions
│   └── web-scraped/        # Custom scrapers (ntfy, claude-code-sdk, etc.)
├── scripts/                # All extraction and update tools
├── index.yaml              # Index of all documentation sources
└── AGENTS.md               # Guide for AI agents

Quick Start

Find Documentation

# Search by topic
ls ~/​github/​llm-code-docs/​docs/​llms-txt/ | grep -i react

# Check if a library exists
grep -i "fastapi" ~/​github/​llm-code-docs/​index.yaml

# Read documentation
cat ~/​github/​llm-code-docs/​docs/​llms-txt/​anthropic/​llms-full.txt

Update All Documentation

cd ~/​github/​llm-code-docs
./​scripts/​update.sh

Update Specific Sources

# Update single llms.txt site
python3 scripts/​llms-txt-scraper.py --site anthropic

# Force re-download (ignore cache)
python3 scripts/​llms-txt-scraper.py --site vercel-ai-sdk --force

# Update Git repository extractions
python3 scripts/​extract_docs.py

# Update specific web-scraped docs
python3 scripts/​ntfy-docs.py
python3 scripts/​claude-code-sdk-docs.py

Adding New Documentation

Priority Order

  1. llms.txt - Check if site has llms.txt first (highest quality)
  2. Git repos - For comprehensive docs from source
  3. Web scraping - Last resort for critical docs

Add llms.txt Site

# 1. Check if llms.txt exists (probes common subdomain/​path combinations)
./​scripts/​find-llms-txt.sh example.com
# Or manually check:
curl -sL https://docs.example.com/llms.txt | head -20

# 2. Add to config (alphabetical order)
# Edit scripts/​llms-sites.yaml:
#   - name: example-site
#     base_url: https://docs.example.com/
#     description: Example documentation

# 3. Download
python3 scripts/​llms-txt-scraper.py --site example-site

# 4. Verify
ls -la docs/​llms-txt/​example-site/

# 5. Update index
python3 scripts/​update-index.py

Add Web Scraper (No llms.txt)

# 1. Create scraper script based on existing pattern
cp scripts/​ntfy-docs.py scripts/​newsite-docs.py

# 2. Edit script:
#    - Update BASE_URL
#    - Update DOC_PAGES list (from sitemap or manual)
#    - Update output path to docs/​web-scraped/​newsite/

# 3. Run scraper
python3 scripts/​newsite-docs.py

# 4. Update index
python3 scripts/​update-index.py

Add Git Repository

# Edit scripts/​repo_config.yaml:
repositories:
  - name: example
    repo_url: https://github.com/owner/repo
    source_folder: docs/
    target_folder: docs/​github-scraped/​example
    branch: main

# Run extraction
python3 scripts/​extract_docs.py

Key Scripts

ScriptPurpose
find-llms-txt.shProbe domain for llms.txt (checks subdomains/​paths)
llms-txt-scraper.pyDownload from 228+ llms.txt sites
extract_docs.pyClone and extract Git repositories
claude-code-sdk-docs.pyClaude Code SDK documentation
ntfy-docs.pyntfy push notification docs
update-index.pyUpdate index.yaml with current state
update.shMaster script runs all updates

Configuration Files

llms-sites.yaml

sites:
- name: anthropic
  base_url: https://docs.anthropic.com/
  description: Claude AI documentation
- name: vercel-ai-sdk
  base_url: https://ai-sdk.dev/
  description: AI SDK for building AI-powered apps

repo_config.yaml

repositories:
  - name: fastapi
    repo_url: https://github.com/fastapi/fastapi
    source_folder: docs/​en/​docs/
    target_folder: docs/​github-scraped/​fastapi
    branch: master

Caching System

  • Cache duration: 23 hours
  • Files downloaded within cache window are skipped
  • Use --force to override cache
  • Parallel downloads with 15 concurrent workers

Common Tasks

Check What's Available

# List all llms.txt sites
ls docs/​llms-txt/ | wc -l

# List web-scraped docs
ls docs/​web-scraped/

# Search for specific topic
grep -r "authentication" docs/​llms-txt/​*/​llms-full.txt | head -20

Verify Documentation Quality

# Check file sizes (small = possibly incomplete)
ls -lh docs/​llms-txt/​*/​llms-full.txt | sort -k5 -h | tail -10

# Check for code examples
grep -l '```' docs/​llms-txt/​anthropic/​*.md | wc -l

# Verify source headers present
head -5 docs/​web-scraped/​ntfy/​publish.md

Troubleshooting

Site returns 404
# Probe all common locations automatically
./​scripts/​find-llms-txt.sh example.com

# Or check manually
curl -sI https://docs.example.com/llms.txt
curl -sI https://example.com/llms.txt
curl -sI https://docs.example.com/llms-full.txt
Push blocked by secrets
# Find and redact test credentials
grep -n "sk_test\|AC[0-9a-f]\{32\}" docs/​web-scraped/​*/

# Replace with placeholders
sed -i '' 's/​AC12345.../​AC_EXAMPLE_SID/​g' file.md

Statistics

  • 228 llms.txt sites fetched
  • 14 Git repositories extracted
  • 3 web-scraped documentation sets
  • 12,000+ markdown files
  • 300MB+ total documentation

See Also