google-agents-cli-scaffold
Project scaffolding, deployment configuration, and CI/CD setup for Google ADK agents.
data-pipeline-monitoring
Monitor and troubleshoot dual-pipeline data collection systems on GCP. This skill should be used when checking pipeline health, viewing logs, diagnosing failures, or monitoring long-running operations for data collection workflows. Supports Cloud Run Jobs (batch pipelines) and VM systemd services...
Full skill instructions
Monitor and operate dual-pipeline data collection systems deployed on Google Cloud Platform.
Provide systematic workflows for:
Invoke this skill when the user mentions any of:
This skill applies to dual-pipeline architectures where:
Use the provided health check script for automated status verification:
python3 scripts/check_pipeline_health.py \
--gcp-project PROJECT_ID \
--cloud-run-job JOB_NAME \
--region REGION \
--vm-name VM_NAME \
--vm-zone ZONE \
--systemd-service SERVICE_NAME
Output: Status report showing OK/WARNING/CRITICAL for each component
Manual verification (if script unavailable):
Check batch pipeline last execution:
gcloud run jobs executions list \
--job JOB_NAME \
--region REGION \
--project PROJECT_ID \
--limit 1 \
--format "value(metadata.name,status.conditions[0].type)"
Check real-time pipeline service status:
gcloud compute ssh VM_NAME \
--zone ZONE \
--project PROJECT_ID \
--command='sudo systemctl is-active SERVICE_NAME'
Expected: Batch shows Completed, real-time shows active
Use the provided log viewer script for unified log access:
# Cloud Run Job logs
bash scripts/view_logs.sh \
--type cloud-run \
--project PROJECT_ID \
--job JOB_NAME \
--region REGION \
--lines 50
# VM systemd service logs
bash scripts/view_logs.sh \
--type systemd \
--project PROJECT_ID \
--vm VM_NAME \
--zone ZONE \
--service SERVICE_NAME \
--lines 50 \
--follow
Common filters:
--filter "ERROR" - Show only errors--filter "Block [0-9]+" - Show block collection progress--follow or -f - Real-time log streamingManual commands (see references/gcp-monitoring-patterns.md for complete patterns):
For Cloud Run:
gcloud logging read \
"resource.type=cloud_run_job AND resource.labels.job_name=JOB_NAME" \
--limit 50 \
--project PROJECT_ID
For VM systemd:
gcloud compute ssh VM_NAME \
--zone ZONE \
--project PROJECT_ID \
--command='sudo journalctl -u SERVICE_NAME -n 50'
Step 1: Identify the failing component
Run health check (Workflow 1) to determine which pipeline is failing.
Step 2: View recent logs
Run log viewer (Workflow 2) for the failing component, focusing on ERROR severity.
Step 3: Consult troubleshooting guide
Read references/troubleshooting-guide.md for common failure modes matching the error symptoms.
Common failure patterns:
.strip()Step 4: Apply recovery procedure
Execute the recovery commands from the troubleshooting guide.
Step 5: Verify resolution
Re-run health check to confirm both pipelines return to OK status.
For operations like historical backfills that run for hours:
Start operation in background:
gcloud compute ssh VM_NAME \
--zone ZONE \
--project PROJECT_ID \
--command='
cd ~/workdir
nohup python3 script.py > progress.log 2>&1 &
echo "Started with PID: $!"
'
Check progress (repeat periodically):
gcloud compute ssh VM_NAME \
--zone ZONE \
--project PROJECT_ID \
--command='tail -50 ~/workdir/progress.log'
Check if process still running:
gcloud compute ssh VM_NAME \
--zone ZONE \
--project PROJECT_ID \
--command='ps aux | grep script.py | grep -v grep'
For Cloud Run Jobs:
Manually trigger a new execution:
gcloud run jobs execute JOB_NAME \
--region REGION \
--project PROJECT_ID
For VM systemd services:
Restart the service:
gcloud compute ssh VM_NAME \
--zone ZONE \
--project PROJECT_ID \
--command='sudo systemctl restart SERVICE_NAME'
Wait 5 seconds, then verify:
gcloud compute ssh VM_NAME \
--zone ZONE \
--project PROJECT_ID \
--command='sudo systemctl status SERVICE_NAME --no-pager'
When code changes are needed to resolve issues:
Step 1: Update the code locally (use Edit tool)
Step 2: Copy updated file to VM:
gcloud compute scp LOCAL_FILE VM_NAME:REMOTE_PATH \
--zone ZONE \
--project PROJECT_ID
Step 3: Restart service to apply changes:
gcloud compute ssh VM_NAME \
--zone ZONE \
--project PROJECT_ID \
--command='sudo systemctl restart SERVICE_NAME'
Step 4: Verify fix by checking logs:
gcloud compute ssh VM_NAME \
--zone ZONE \
--project PROJECT_ID \
--command='sudo journalctl -u SERVICE_NAME -n 20 --no-pager'
scripts/check_pipeline_health.py - Automated health check for both pipelines
scripts/view_logs.sh - Unified log viewer for Cloud Run and systemd
--follow)--filter "PATTERN")references/gcp-monitoring-patterns.md - Complete command reference
references/troubleshooting-guide.md - Failure diagnosis and recovery
To use the scripts, provide these parameters:
GCP Configuration:
PROJECT_ID - GCP project IDREGION - Cloud Run region (e.g., us-east1)Batch Pipeline:
JOB_NAME - Cloud Run Job nameReal-Time Pipeline:
VM_NAME - VM instance nameZONE - VM zone (e.g., us-east1-b)SERVICE_NAME - systemd service nameExample values from MotherDuck integration:
PROJECT_ID=eonlabs-ethereum-bq
REGION=us-east1
JOB_NAME=eth-md-updater
VM_NAME=eth-realtime-collector
ZONE=us-east1-b
SERVICE_NAME=eth-collector
This skill complements project-specific documentation:
Load project documentation when needed for context, but use this skill's workflows for operational tasks.
Important: This skill monitors pipeline health (whether pipelines are running), NOT data completeness.
For verifying actual data in ClickHouse (block counts, historical data presence):
scripts/clickhouse/verify_blocks.py for database state verificationCommon scenario: Pipeline health checks show OK, but historical data is missing. This happens because:
See historical-backfill-execution skill for backfill operations and troubleshooting missing data.
Project scaffolding, deployment configuration, and CI/CD setup for Google ADK agents.
Set up tracing, logging, and monitoring for deployed ADK agents across Cloud Trace, BigQuery, and third-party platforms.
Enterprise Azure infrastructure architect generating Bicep or Terraform from workload descriptions.
Plan and configure production-ready Azure Kubernetes Service clusters with Day-0 and Day-1 best practices.
Raw mechanical interfaces fusing Swiss typographic print with military terminal aesthetics. Rigid grids, extreme type scale contrast, utilitarian color, analog degradation effects. For data-heavy dashboards, portfolios, or editorial sites that need to feel like declassified blueprints.
Web search, scraping, extraction, crawling, and monitoring via ScrapeGraph AI CLI.
Skill for working with Firebase Hosting (Classic). Use this when you want to deploy static web apps, Single Page Apps (SPAs), or simple microservices. Do NOT use for Firebase App Hosting.
Deploy and manage web apps with Firebase App Hosting. Use this skill when deploying Next.js/Angular apps with backends.
Deploy applications and websites to Vercel. Use when the user requests deployment actions like "deploy my app", "deploy and give me the link", "push this live", or "create a preview deployment".
Build SEO-optimized pages at scale using templates, data, and proven playbook patterns.
Design and build isolated, reusable Convex backend components with clear boundaries and app-facing wrappers.
Deploy and manage projects on Vercel using token-based authentication. Use when working with Vercel CLI using access tokens rather than interactive login — e.g. "deploy to vercel", "set up vercel", "add environment variables to vercel".
Opus.ai: Revolutionize Your Web Experience
Build a no-code AI app in minutes.
An IDE for code migration from legacy to modern frameworks through coding agents.
Automate CGI animation in live-action scenes
Branded artistic QR-code concepts
Launch a website in seconds with AI.
Streamline Your Coding Experience with AI Code Helper
Convert any screenshot or design to clean code.