llms-txt-crawler
Fetch and crawl llms.txt files from websites. Parses the llms.txt format to extract page URLs and downloads all listed content. Use when you need to gather documentation or content from a website that provides an llms.txt file.
Full skill instructions
This skill enables you to fetch llms.txt files from websites and crawl all pages listed within them. The llms.txt format is a standard way for websites to provide LLM-friendly content listings.
The llms.txt file typically follows this format:
# Site Name
## Section Name
- [Page Title](https://example.com/page.md): Description of the page
- [Another Page](https://example.com/another.md): Another description
This skill parses these files and downloads all linked content.
Run the crawl script with a target URL:
cd /path/to/skills/llms-txt-crawler/scripts
npm install # First time only
node crawl.js --url https://example.com
| Option | Short | Description | Default |
|---|---|---|---|
--url | -u | Base URL of the site with llms.txt | Required |
--output | -o | Output directory for crawled files | ./output |
--format | -f | Output format: md, json, or txt | md |
--delay | -d | Delay between requests in milliseconds | 500 |
--concurrent | -c | Maximum concurrent requests | 3 |
Crawl agentskills.io documentation:
node crawl.js --url https://agentskills.io --output ./agentskills-docs
Crawl with custom rate limiting:
node crawl.js --url https://example.com --delay 1000 --concurrent 2
Output as JSON:
node crawl.js --url https://example.com --format json
The script creates the following output structure:
output/
├── llms.txt # Original llms.txt file
├── index.json # Metadata about all crawled pages
└── pages/
├── page-1.md
├── page-2.md
└── ...
When using this skill in an agent workflow:
index.json file contains metadata about all pages
Comprehensive DevOps and deployment workflow that orchestrates infrastructure automation, CI/CD pipelines, container orchestration, and cloud deployment. Handles everything from infrastructure as code and pipeline setup to monitoring, scaling, and disaster recovery.
Integrate avnu SDK for Starknet DeFi: token swaps with best-price routing, DCA recurring buys, native staking, gasless transactions via paymaster, and market data. Use when building Starknet trading, DEX aggregation, or DeFi automation features.
发布内容和文章到 X (Twitter)。支持带图片的普通推文和 X Articles（长文）。使用真实的 Chrome 和 CDP 来绕过自动化检测。
Master orchestrator for generating Ralph Wiggum-compatible prompts. Analyzes task requirements and routes to appropriate generator (single-task, multi-task, project, or research). Use when you need to create any Ralph loop prompt and want automatic selection of the right generator.
Use when the user asks to compare branches, analyze git diffs, review changes between branches, update specifications based on code changes, or analyze what changed. Triggers on keywords like "compare branches", "git diff", "what changed", "branch comparison", "code changes", "spec update".
Build high-performance FastAPI applications with async routes, validation, dependency injection, security, and automatic API documentation. Use when developing modern Python APIs with async support, automatic OpenAPI documentation, and high performance requirements.
Enables Claude to manage Zendesk Chat conversations, support tickets, and customer service operations
Mine transcripts for knowledge - decisions, learnings, failures, patterns. Triggers: "forge insights", "mine transcripts", "extract knowledge".
Track product analytics and user behavior with Mixpanel's event-based platform.
OCPP protocol reference for EV charging infrastructure development. Covers OCPP 2.0.1 and OCPP 1.6J. Use when working with OCPP messages, charging station code, CSMS/Central System backends, smart charging, transaction handling, or EV charging protocols. Activates on keywords: OCPP, charging stat...
Gather scattered ideas into organized GitHub issues. The bee collects pollen from every flower, checks if the hive already has it, and carefully deposits new work in the comb for later. Never edits code—only organizes the work ahead. Use when creating issues from TODOs or planning work.
Comprehensive best practices for React Native and Expo applications. Contains rules across multiple categories covering performance, animations, UI patterns, and platform-specific optimizations.
Generate, edit, and beat-sync AI video with leading models in one workspace.
The world's fastest calendar for remote work
Transform Your Design with AI Designer by ImgCreator.ai
Revolutionizing Video Production with AI-Powered Creativity
Extend an image past the frame and let AI fill the new aspect ratio.
Discover your celebrity doppelgänger with StarByFace!
ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.
Opus.ai: Revolutionize Your Web Experience