xiaohongshu
xiaohongshu
Techniques and tools for ensuring the accuracy, completeness, and reliability of data across the pipeline.
Full skill instructions
Data Quality (DQ) Monitoring is the continuous process of validating data against predefined rules and expectations. In a modern data stack, monitoring must happen at every stage: Ingestion, Transformation, and Serving.
Core Principle: "Garbage in, garbage out. If the data is wrong, the analytics, AI, and decisions will be wrong too."
Common industry standards for measuring data health:
| Dimension | Definition | Example Metric |
|---|---|---|
| Accuracy | Does the data reflect reality? | % of records matching the source system. |
| Completeness | Are there missing values? | % of non-null values in mandatory fields. |
| Consistency | Does data match across systems? | Do user names match in both SQL and Redis? |
| Timeliness | Is data fresh enough? | Data latency (current time - event time). |
| Uniqueness | Are there duplicate records? | Count of duplicate primary keys. |
| Validity | Does it follow specified formats? | % of emails following the regex pattern. |
Ideal for SQL-based transformations.
# schema.yml
models:
- name: active_users
columns:
- name: user_id
tests:
- unique
- not_null
- name: age
tests:
- accepted_values:
values: [18, 120]
The most flexible tool for complex Python-based data pipelines.
import great_expectations as ge
df = ge.read_csv("data.csv")
df.expect_column_values_to_be_between("age", 18, 120)
df.expect_column_values_to_not_be_null("email")
df.expect_column_values_to_match_regex("email", r"[\w.-]+@[\w.-]+")
You can run "Check" queries alongside your production workload to catch silent failures.
-- Detect "Price Drift" anomaly
SELECT
product_id,
AVG(price) OVER(PARTITION BY category) as avg_cat_price,
price
FROM current_inventory
WHERE price > (avg_cat_price * 5) -- Alert if price is 5x the average
| Tool | Focus | Best For |
|---|---|---|
| Great Expectations | Validation | Python pipelines, Airflow integration. |
| Monte Carlo | Data Observability | ML-driven anomaly detection, lineage. |
| Soda | Data Contracts / Monitoring | Collaborative DQ for Data & Business teams. |
| Anodot | Streaming Anomaly | Catching spikes/dips in real-time metrics. |
Data Drift occurs when the statistical properties of the data change over time, even if the schema stays the same.
When a monitor fails, it should trigger an incident workflow:
A high-level view for stakeholders:
tax_amount to be 0 for all of them.tax_amount > 0 for all non-exempt regions.NULL values in primary keys?43-data-reliability/data-contracts43-data-reliability/data-lineage42-cost-engineering/cost-observabilityxiaohongshu
technical spec
product ux expert
database patterns
Conduct multi-agent task orchestration and workflow coordination.
Initialize project with Conductor artifacts (product definition,
Expert in web animations, transitions, and motion design using Framer Motion and CSS
Creates Mermaid and ASCII diagrams for flowcharts, architecture, ERDs, state machines, mindmaps, and more. Use when user mentions diagram, flowchart, mermaid, ASCII diagram, text diagram, terminal diagram, visualize, C4, mindmap, architecture diagram, sequence diagram, ERD, or needs visual docume...
PostgreSQL bindings for H3 hexagonal grid system. Use when working with H3 cells in Postgres, including spatial indexing, geometry/geography integration, and raster analysis.
Context-Driven Development skill for projects using Conductor. Use this skill when you detect a `conductor/` directory in the project, when working on tasks defined in a `plan.md` file, or when the user asks about tracks, specs, or plans. Automatically applies TDD workflow, tracks task completion...
Display project status, active tracks, and next actions
Official Stakpak application containerization standard operating procedure, a step-by-step guidline to properly dockerize applications. This is a rule book curated by the Stakpak Team.
Generate, edit, and beat-sync AI video with leading models in one workspace.
The world's fastest calendar for remote work
Transform Your Design with AI Designer by ImgCreator.ai
Revolutionizing Video Production with AI-Powered Creativity
Extend an image past the frame and let AI fill the new aspect ratio.
Discover your celebrity doppelgänger with StarByFace!
ChainClarity explains 700+ crypto whitepapers in plain English, with layered summaries, comparisons, research tools, alerts, and a $4.99 Pro plan.
Opus.ai: Revolutionize Your Web Experience