Overview
The Quality Agent runs automated data quality checks across your data estate using Great Expectations. It profiles your data, generates intelligent quality expectations based on column types and distributions, and executes validations — surfacing issues proactively so problems are caught before they reach your reports and dashboards.What You Can Ask
- “Check the quality of our customer table”
- “Are there any null values in the orders dataset?”
- “Run a freshness check on all our staging tables”
- “What’s the data quality score for our sales data?”
- “Find duplicate records in the user profiles table”
- “Validate the web analytics dataset”
How It Works
- Analyze dataset structure — The agent queries your warehouse for a data sample (up to 5,000 rows) and creates a comprehensive profile — column types, null percentages, unique counts, value distributions, and statistical summaries.
- Generate quality expectations — Based on the data profile, an LLM generates a tailored set of quality checks — typically 10–25 high-confidence expectations covering completeness, accuracy, consistency, and more.
- You select which checks to run — The proposed expectations are presented for your review. You choose which ones to execute — keeping you in control of how quality is measured.
- Run quality validation — Selected expectations are executed against your data using Great Expectations. Results include per-column pass/fail status, detailed statistics, and an overall quality score.
Quality Dimensions
Completeness
Are required fields populated? What percentage of values are missing or null? Checks null rates against configurable thresholds.
Accuracy
Do values match expected patterns, ranges, and business rules? Validates formats (emails, phones), statistical bounds, and value domains.
Consistency
Is the data coherent across columns and tables? Checks referential integrity, cross-column relationships, and value set membership.
Freshness
How recently was the data updated? Is it within expected SLA windows? Monitors timestamps and update frequency.
Supported Quality Checks
The Quality Agent supports over 50 types of data quality expectations, organized by category:| Category | Example Checks |
|---|---|
| Completeness | Null value detection, non-null proportion thresholds |
| Uniqueness | Duplicate detection, compound column uniqueness |
| Value Ranges | Min/max bounds, mean/median ranges, z-score outlier detection |
| Pattern Matching | Regex validation for emails, phones, and custom formats |
| Set Membership | Value-in-set checks, distinct value set validation |
| Schema | Column count, column ordering, column type verification |
| Cross-Column | Pair comparisons (A > B), multi-column sum validation |
| Distribution | Statistical bounds on mean, median, standard deviation |
Quality Reports
Every validation produces a detailed quality report:- Dataset Overview — Total rows, columns, and overall missing value percentage
- Column Analysis — Per-column breakdown showing type, null rate, unique count, and quality status
- Failed Expectations — Specific details on what failed and why, with counts and examples
- Overall Quality Score — Percentage of checks passed, stored as a trackable metric
- Pass — All checks for the column passed
- Warning — Some checks passed, some failed
- Fail — Critical checks failed for the column
Human-in-the-Loop
The Quality Agent includes a human-in-the-loop step before executing validations:- The agent profiles your data and generates recommended expectations
- You review the proposed checks — each with severity, confidence, and description
- You select which expectations to run
- Only approved checks are executed
Data Connections
The Quality Agent accesses data through the platform’s secure infrastructure:Redshift Warehouse
Queries your dedicated Redshift cluster for data samples via cross-account IAM roles. Supports fully qualified table names with schema isolation.
OpenMetadata Catalog
Reads asset metadata to identify tables, resolve names, and record quality execution results back to the catalog.
S3 Storage
Quality reports are stored as formatted documents in your organization’s S3 storage for long-term access and auditing.
Great Expectations
Validations execute in an ephemeral environment — no persistent configuration required. Each run is isolated and stateless.
Works With Other Agents
- Governance Agent — Quality checks are part of the governance workflow. Quality scores feed into compliance reporting and data lifecycle tracking.
- Analyst Agent — Quality issues are flagged before analysis begins so you know your confidence level in the underlying data.
- Engineering Agent — Quality checks validate transformation outputs after dbt models run, ensuring transformations produce correct results.
- Metadata Agent — Quality scores and execution history are stored as metadata on data assets in your catalog.
The Quality Agent is part of the BrightAgent architecture. See capabilities for the full list of what BrightAgent can do.