Video Ingest — via Inngest Pipeline

Video Ingest — via Inngest Pipeline

Videos are ingested through the Inngest event bus. Do not run yt-dlp, mlx-whisper, scp, or create vault notes manually. The pipeline handles everything: download → NAS transfer → transcription → vault note → summary enrichment.

Quick Start

joelclaw send pipeline/​video.requested -d '{"url":"URL_HERE"}'

# legacy alias still accepted
joelclaw send pipeline/​video.download -d '{"url":"URL_HERE"}'

That's it. The event chain handles the rest.

Alternative (raw curl):

curl -s -X POST "http://localhost:8288/​e/​37aa349b89692d657d276a40e0e47a15" \
  -H "Content-Type: application/​json" \
  -d '{"name":"pipeline/​video.requested","data":{"url":"URL_HERE"}}'

Pipeline Flow

pipeline/​video.requested       — you send this (legacy alias: pipeline/​video.download)
    ↓
video-download function        — yt-dlp → /​tmp → NAS transfer
    ↓ emits
pipeline/​video.downloaded      — logged by system-logger
pipeline/​transcript.requested  — auto-triggered (legacy alias: pipeline/​transcript.process)
    ↓
transcript-process function    — mlx-whisper (M4 Pro, ~5min/​hr of video)
    ↓ emits
pipeline/​transcript.processed  — logged
content/​summarize.requested    — auto-triggered (legacy alias: content/​summarize)
    ↓
content-summarize function     — pi enrichment → vault note with summary
    ↓ emits
content/​summarized             — logged, done

Before Sending: Health Check

Always verify the pipeline is healthy before sending events:

# Inngest server
curl -s http://localhost:8288/​health

# Worker (should show functions including video-download)
curl -s http://localhost:3111/ | python3 -c "
import json,sys
d=json.load(sys.stdin)
fns = [f.get('id','?') for f in d.get('functions',[])]
print(f'Worker OK — {len(fns)} functions: {', '.join(fns)}')
"

# Docker container running
docker ps --filter ancestor=inngest/​inngest --format "table {{.Status}}\t{{.Ports}}"

If the worker is down:

kubectl -n joelclaw rollout restart deployment/​system-bus-worker
kubectl -n joelclaw rollout status deployment/​system-bus-worker --timeout=180s
joelclaw refresh

Monitoring a Run

Watch progress in real-time

# Worker logs — shows step execution + failures
kubectl logs -n joelclaw deploy/​system-bus-worker -f

# Docker logs — shows event dispatch
docker logs -f $(docker ps -q --filter ancestor=inngest/​inngest) 2>&1 | grep -v DEBUG

Check if events fired

# Look for the video's events in Docker logs
docker logs $(docker ps -q --filter ancestor=inngest/​inngest) 2>&1 | grep -i "video\|transcript\|summarize" | tail -20

Dashboard

Open http://localhost:8288 in browser — shows functions, events, runs with per-step traces.

Verify completion

# Check if vault note was created
ls -la ~/​Vault/​Resources/​videos/​*SLUG*

# Check system log for pipeline entries
tail -10 ~/​Vault/​system/​system-log.jsonl | grep -i video

Batch Ingest

Send multiple events. Inngest queues and processes them with concurrency control:

joelclaw send pipeline/​video.requested -d '{"url":"https://youtube.com/​watch?v=XXXX"}'
joelclaw send pipeline/​video.requested -d '{"url":"https://youtube.com/​watch?v=YYYY"}'
joelclaw send pipeline/​video.requested -d '{"url":"https://youtube.com/​watch?v=ZZZZ"}'

Manual Transcript (Non-YouTube)

For audio files already on disk, or raw text from Granola/​Fathom:

# From audio file
joelclaw send pipeline/​transcript.requested -d '{"source":"manual","audioPath":"/​path/​to/​audio.mp4","title":"Recording Title","slug":"recording-title"}'

# From raw text (Granola, Fathom, etc.)
joelclaw send pipeline/​transcript.requested -d '{"source":"granola","text":"transcript text...","title":"Meeting Title","slug":"meeting-title"}'

Re-run Summary Only

If the vault note exists but needs a better summary:

joelclaw send content/​summarize.requested -d '{"vaultPath":"/​Users/​joel/​Vault/​Resources/​videos/​SLUG.md"}'

Options

Where Things End Up

Troubleshooting

If events are accepted (200 OK) but nothing happens:

1. Check Docker→worker connectivity — the most common issue:

   docker logs $(docker ps -q --filter ancestor=inngest/​inngest) 2>&1 | grep -E "ERROR|Unable" | tail -5
   

   If you see "Unable to reach SDK URL" → see the inngest skill's serveHost gotcha.

2. Check worker is actually running:

   kubectl get deploy -n joelclaw system-bus-worker
   kubectl get pods -n joelclaw -l app=system-bus-worker
   

3. Check worker errors for the specific function:

   kubectl logs -n joelclaw deploy/​system-bus-worker --tail=80
   

4. Use inngest-debug skill for deep inspection of specific run IDs via GraphQL.

What NOT to Do

• ❌ Don't run yt-dlp directly — the pipeline handles download + NAS transfer
• ❌ Don't run mlx_whisper directly — the pipeline handles transcription
• ❌ Don't scp to NAS manually — the pipeline handles transfer
• ❌ Don't create vault notes manually — the pipeline creates them with proper frontmatter
• ❌ Don't use codex/​background tasks for video processing — Inngest is durable and has retries