Blog Scraper Skill

Blog Scraper Skill

Overview

This skill fetches blog articles from tech-lab.sios.jp/​archives/​*, compresses the HTML content by removing unnecessary attributes and whitespace, and saves the result to the doc/ directory with metadata.

When to Use

• User requests to fetch a specific blog article
• User wants to update existing cached articles
• User needs to scrape multiple articles for analysis or documentation

Usage

Single Article

URL=https://tech-lab.sios.jp/​archives/[article-id] npm run scraper

Example:

URL=https://tech-lab.sios.jp/​archives/​48397 npm run scraper

Multiple Articles

For multiple articles, run the command sequentially for each URL.

Output

The scraper will:

1. Fetch and parse the HTML from the specified URL
2. Extract content using the CSS selector section.entry-content
3. Compress by removing:
   • Scripts, styles, and noscript tags
   • Class, ID, and style attributes
   • Whitespace between tags
4. Preserve:
   • Image alt text as [画像: alt]
   • Image src URLs
   • Link href attributes
5. Add metadata as HTML comment:
   • OGP title
   • Source URL
   • OGP image URL
   • Extraction timestamp
6. Save to docs/​data/​tech-lab-sios-jp-archives-[id].html
7. Report compression statistics:
   • Token count reduction (estimated for Claude)
   • Compression ratio percentages
   • File size

Cache Behavior

• If the target HTML file already exists in docs/​data/, the scraper skips fetching and reports the existing file size
• To re-fetch, delete the existing HTML file first

Token Estimation

The scraper estimates Claude token usage for Japanese content:

• Hiragana/​Katakana: ~1.5 chars/​token
• Kanji: ~1 char/​token
• ASCII: ~4 chars/​token
• Other: ~2 chars/​token

Typical compression achieves 60-85% token reduction.

Implementation Details

See application/​tools/​scraper.ts for the TypeScript implementation using:

• node-fetch for HTTP requests
• cheerio for HTML parsing
• OGP metadata extraction
• Custom token estimation for Japanese text

Permissions Required

This skill requires the following permissions in .claude/​settings.local.json:

{
  "permissions": {
    "allow": [
      "Bash(npm run scraper:*)",
      "Bash(URL=:*)"
    ]
  }
}

Note: The Bash(URL=:*) permission uses prefix matching to allow any URL environment variable pattern. This is a broad permission - consider restricting to specific domains if needed for security.