Cost Optimization Skill

Cost Optimization Skill

Use this skill when investigating or reducing Coralogix data costs. It covers the full cost management lifecycle: measuring current spend, reviewing TCO policies, adjusting retention periods, and configuring archive storage for cold data.

---

CLI Commands

Key flags:

• All commands support -o json for structured output and -p <profile> for profile selection
• cx usage daily accepts --type processed-gbs|units|evaluation-tokens and --start/--end time filters
• cx usage summary accepts --start/--end time filters
• cx usage logs-count and cx usage spans-count accept --start/--end time filters, defaulting to the last 24h, plus --resolution (default 1h), --subsystem-aggregation, --application-aggregation, and repeated --param KEY=VALUE for API filter query params
• Data usage summary and count endpoints are documented as newline-delimited JSON over Accept: text/​event-stream; the CLI handles that transport and normalizes count chunks into .result.logsCount[] or .result.spansCount[].
• cx tco create/​update, cx retentions update, cx archive logs set, cx archive metrics create/​update/​validate use --from-file <path> (or - for stdin)

---

Cost Investigation Workflow

Follow these steps to diagnose and reduce costs:

Step 1: Measure Current Usage

cx usage summary -o json
cx usage summary --start now-30d -o json
cx usage daily --type processed-gbs --start now-7d -o json
cx usage logs-count --start now-7d --end now -o json
cx usage spans-count --start now-7d --end now -o json

Identify which data types consume the most volume. Use jq to sort:

cx usage summary -o json | jq '[.[] | {name, daily_avg: .avg_daily_gb}] | sort_by(.daily_avg) | reverse'

Step 2: Review TCO Policies

cx tco list -o json
cx tco settings -o json

TCO policies control which logs go to Frequent Search (expensive, fast) vs. Archive (cheap, slower). Check if high-volume, low-value logs are on Frequent Search:

cx tco list -o json | jq '.[] | select(.priority == "LOW") | {name, application, subsystem, archive_retention}'

Step 3: Check Retention Settings

cx retentions list -o json
cx retentions status -o json

Long retention periods increase storage costs. Identify indices with unnecessarily long retention.

Step 4: Check Archive Configuration

cx archive logs get -o json
cx archive metrics get -o json

Verify that archive storage is configured for cold data. If no archive is set up, that's a cost-saving opportunity.

Step 5: Recommend Optimizations

Based on findings, recommend changes in priority order (highest impact first).

---

Common Optimization Patterns

---

jq Examples

Usage Analysis

# Top consumers by daily volume
cx usage summary -o json | jq '[.[] | {name, daily_avg: .avg_daily_gb}] | sort_by(.daily_avg) | reverse | .[0:10]'

# Daily trend for the past week
cx usage daily --type processed-gbs --start now-7d -o json | jq '[.[] | {date, gb: .processed_gbs}]'

# Total logs and spans counts
cx usage logs-count --start now-7d --end now -o json | jq '[.result.logsCount[]?.logsCount | tonumber] | add // 0'
cx usage spans-count --start now-7d --end now -o json | jq '[.result.spansCount[]? | ((.successSpanCount | tonumber) + (.errorSpanCount | tonumber) + (.lowSuccessSpanCount | tonumber) + (.lowErrorSpanCount | tonumber) + (.mediumSuccessSpanCount | tonumber) + (.mediumErrorSpanCount | tonumber))] | add // 0'

TCO Policy Analysis

# Policies routing to archive tier
cx tco list -o json | jq '[.[] | select(.archive_retention != null)]'

# Policies by priority
cx tco list -o json | jq 'group_by(.priority) | map({priority: .[0].priority, count: length})'

# Test if a log pattern matches a policy
cx tco test --from-file test-definition.json -o json

Retention Review

# All retention settings
cx retentions list -o json | jq '.[]'

# Check if retention is active
cx retentions status -o json

Archive Status

# Logs archive configuration
cx archive logs get -o json | jq '{active: .active, bucket: .bucket}'

# Metrics archive configuration
cx archive metrics get -o json | jq '{enabled: .enabled, bucket: .bucket}'

---

Applying Changes

IMPORTANT: NEVER pass --yes without explicit user approval. All write operations across archive, TCO, and retentions require interactive confirmation and the --yes flag to execute non-interactively. Before executing any write operation, describe the exact change to the user and wait for their approval before passing --yes.

Read-only mode: Use --read-only (or CX_READ_ONLY=1) to safely explore cost data without risk of accidental writes. All query commands (usage, tco list/​get, retentions list, archive get) work normally in read-only mode.

Agent mode: When running inside an AI agent, cx fails fast on write operations instead of hanging on a stdin prompt. Get user confirmation first, then re-run with --yes.

When modifying TCO policies, retention, or archive:

1. Template from existing: Get the current configuration as JSON, modify it, then apply:

   cx tco get <policy-id> -o json > policy.json
   # Edit policy.json
   cx tco update --from-file policy.json
   

2. Verify after changes: Re-run the diagnosis commands to confirm the change took effect.

3. TCO policy ordering matters: Use cx tco reorder --from-file to set priority order. Policies are evaluated top-to-bottom; the first match wins.

---

Metrics-Based Cost Analysis

The cx usage API gives summaries, but for billing-accurate analysis, anomaly detection, and breakdown by pillar/​feature, query the customer metrics exporter via PromQL.

Key Metrics

Concept-to-Metric Mapping

• Billing / plan usage / consumption -> cx_data_usage_units + cx_data_plan_units_per_day
• Processed bytes / data volume -> cx_data_usage_total
• AI evaluation tokens -> cx_data_usage_tokens_total
• Metric samples -> cx_data_usage_samples_total
• Overage / PAYG -> cx_data_usage_payg_units

Common PromQL Queries

# Today's billable units consumed so far
cx metrics query 'sum(cx_data_usage_units)' --time now -o json

# Units breakdown by pillar
cx metrics query 'sum by (pillar) (cx_data_usage_units)' --time now -o json

# Daily plan quota
cx metrics query 'cx_data_plan_units_per_day' --time now -o json

# Plan consumption percentage
cx metrics query '100 * sum(cx_data_usage_units) / cx_data_plan_units_per_day' --time now -o json

# Units by feature group
cx metrics query 'sum by (feature_group_id) (cx_data_usage_units)' --time now -o json

# PAYG overage (if any)
cx metrics query 'cx_data_usage_payg_units' --time now -o json

UTC-Day Bucketing Rules

All usage metrics accumulate from UTC midnight and reset at 00:00 UTC:

• An instant query during the day returns "today so far"
• For completed-day totals, use the last sample before midnight
• Never subtract values across a UTC midnight boundary
• For weekly/​monthly analysis, derive completed daily totals first, then roll up
• Exclude the current partial UTC day when computing trends or averages

Anomaly Detection

When investigating usage anomalies:

1. Compare completed UTC days (exclude current partial day)
2. Break down by: measurement_type -> pillar -> entity_type -> priority -> feature_group_id -> application_name -> subsystem_name
3. Prefer same-weekday comparisons for seasonal traffic
4. Use cx_data_usage_units for billing anomalies, cx_data_usage_total for volume anomalies

Breakdown Labels

Usage metrics support these grouping dimensions: pillar, entity_type, priority, measurement_type, feature_group_id, feature_id, application_name, subsystem_name.

---

Key Principles

• Measure before changing - always run usage/​summary commands before modifying policies
• Use -o json with jq - structured output enables precise analysis
• Verify changes - re-query after every modification to confirm it took effect
• Multi-profile awareness - use -p <profile> or --all-profiles to compare costs across environments
• Template from existing - get current config as JSON before creating or updating
• TCO is the biggest lever - moving logs from Frequent Search to Archive tier has the largest cost impact

---

Related Skills

• cx-telemetry-querying - investigate what data is being ingested (query logs, metrics, and spans to identify high-volume sources)