# BrightAgent Architecture
Source: https://docs.brighthive.io/brightagent/architecture
A multi-agent AI system built on LangGraph — how the BrightAgent orchestrates specialized agents, accesses data, and maintains quality.
## System Overview
BrightAgent is a multi-agent AI system built on **LangGraph** that handles end-to-end data operations through natural language. BrightAgent orchestrates specialized agents, each focused on a specific domain of the data lifecycle. Agents access data through the platform's secure infrastructure — never directly.
![BrightAgent Architecture Diagram]()
## BrightAgent (Orchestrator)
The BrightAgent is the central coordinator. Every user query flows through it:
```mermaid theme={null}
graph TD
A[User Query] --> B[BrightAgent]
B --> C["Intent Classification"]
C --> D{"Route to Agent(s)"}
D --> E[Retrieval Agent]
D --> F[Analyst Agent]
D --> G[Visualization Agent]
D --> H[Engineering Agent]
D --> I[Governance Agent]
D --> J[Quality Agent]
D --> K[Metadata Agent]
E & F & G & H & I & J & K --> L[BrightAgent Aggregation]
L --> M["Synthesized Response"]
```
The BrightAgent:
1. **Analyzes intent** — Determines what the user is asking for and which capabilities are needed
2. **Routes to agents** — Selects one or more specialized agents based on the task
3. **Orchestrates workflows** — Coordinates multi-step execution where agents hand off results to each other
4. **Aggregates results** — Combines outputs from all agents into a coherent response
5. **Maintains conversation context** — Tracks state across multi-turn conversations so agents remember what came before
## Specialized Agents
Implements **GraphRAG** (Graph Retrieval Augmented Generation) for intelligent data discovery. Queries Neo4j to find relevant data assets, metadata, and relationships — then surfaces the best matches for the user's query.
Generates and executes **SQL queries** against Redshift. Performs statistical analysis, creates Jupyter notebooks, and produces insights grounded in actual data — not guesses from training data.
Generates **dbt transformation models** with proper SQL, configurations, and tests. Submits everything as a **GitHub PR** for human review — nothing gets deployed without approval.
Creates interactive **Plotly charts** and visualizations. Automatically selects chart types based on data characteristics — bar, line, scatter, pie, heatmap — or follows specific user instructions.
Manages **data quality policies**, compliance rules, and metadata governance. Tracks and maintains **lineage** across the entire data estate via Neo4j.
Runs **data quality checks** — completeness, accuracy, consistency, and freshness — and surfaces issues proactively. Operates as a background agent monitoring data health continuously.
Connects to **OpenMetadata via MCP** to generate descriptions, understand schemas, enrich catalog metadata with tags and documentation, and track data lineage.
Routes Slack messages to BrightAgent, Jira, Notion, Google Drive, and MS Teams via **intent classification** and MCP integrations. Sub-100ms routing latency.
## Data Flow
### How a Query Gets Answered
When a user asks a question, here's what happens end-to-end:
```mermaid theme={null}
graph TD
A["User: 'Show me a chart of sales by region'"] --> B[BrightAgent]
B -->|"Step 1: Classify intent"| C["Needs: retrieval + analysis + visualization"]
C -->|"Step 2: Retrieval"| D[Retrieval Agent]
D -->|"Query Neo4j"| E["Find 'sales' data assets + metadata"]
E -->|"Step 3: Analysis"| F[Analyst Agent]
F -->|"Generate SQL → Execute on Redshift"| G["Aggregate: sales by region"]
G -->|"Step 4: Visualization"| H[Visualization Agent]
H -->|"Generate Plotly chart"| I["Interactive bar chart"]
I -->|"Step 5: Synthesize"| J[BrightAgent]
J --> K["Natural language summary + chart + underlying data"]
```
### How Agents Access Data
Agents never access your data directly. Every query flows through the platform's secure infrastructure:
```mermaid theme={null}
graph LR
A[BrightAgent] --> B["Neo4j (Metadata)"]
A --> C["Platform API (GraphQL)"]
C --> D["Cross-Account IAM"]
D --> E["Redshift (Your Workspace)"]
E --> F["S3 (Your Organization)"]
```
* **Neo4j** provides metadata context — what data exists, where it lives, who owns it, how it relates to other data
* **Redshift** in your dedicated workspace executes queries via cross-account IAM roles
* **S3** in your organization account stores the actual data — Redshift reads it in place via Spectrum
Agents can only access data that the user's workspace is authorized for. No exceptions.
## Agent Coordination
### Parallel Execution
Multiple agents can work simultaneously when tasks are independent. For example, the Retrieval Agent searches for data while the Visualization Agent prepares chart templates — reducing total response time.
### Sequential Chaining
Workflows that depend on prior results run step-by-step: Retrieval finds data → Analyst queries it → Visualization charts the results. Each agent receives the output of the previous step.
### Context Sharing
Agents share relevant context and intermediate results through **LangGraph state**. The Analyst Agent knows exactly which data asset the Retrieval Agent found, including schema, location, and access details.
### Multi-Turn Conversations
The BrightAgent maintains **conversation state** across turns. Users can refine results iteratively:
* *"Show me customer data"* → Retrieval finds datasets
* *"Filter to California only"* → Analyst refines the query using context from the first turn
* *"Chart that as a pie chart"* → Visualization uses the analyst's results
## Human-in-the-Loop
Operations that modify your data infrastructure always require human approval:
| Operation | Approval Mechanism |
| -------------------------- | --------------------------------------------------------------------------- |
| dbt model generation | GitHub PR — your team reviews SQL, tests, and configurations before merging |
| Jupyter notebook execution | Code is presented for review before execution |
| Governance policy changes | Explicit user confirmation required |
| Schema modifications | User must approve before any changes are applied |
This ensures the AI assists your workflow without making irreversible changes autonomously.
## Observability
Every agent interaction is fully traceable:
Full trace visibility into every agent step — from intent classification through tool calls to response synthesis. Includes latency breakdowns, token usage, and error attribution.
Agent invocations, latency (p50/p95/p99), error rates, and token usage tracked via OpenTelemetry for operational dashboards and alerting.
All tool calls, data accessed, SQL generated, and decisions made are logged. Users can inspect exactly what happened behind every response.
Every response is scored for relevance and correctness using DeepEval metrics. Quality trends are tracked across releases to catch regressions early.
## Deployment
* **LangGraph Cloud** — BrightAgent is deployed on LangGraph Cloud for managed orchestration, scaling, and state persistence.
* **MCP Integration** — Model Context Protocol provides validated tool execution and external service connectivity (Jira, Notion, Google Drive, OpenMetadata).
* **LLM Providers** — Powered by OpenAI and Anthropic models, selected per-agent based on task requirements and cost efficiency.
* **Three Environments** — Dev, staging, and production with CI/CD pipelines and evaluation gates before promotion.
## Key Architectural Principles
| Principle | How It's Implemented |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| **Agent-per-Domain** | Each agent specializes in one data domain — retrieval, analysis, engineering, visualization — keeping logic focused and maintainable |
| **Graph-Powered Context** | Neo4j provides rich metadata context for every interaction — lineage, relationships, schema, ownership — via GraphRAG |
| **Secure by Default** | All data access flows through cross-account IAM roles. Agents can only reach data the user's workspace is authorized for |
| **Observable** | Every agent interaction, tool call, and decision is traced via LangSmith and logged for debugging and audit |
| **Human-in-the-Loop** | Irreversible operations require explicit human approval. AI assists — humans decide |
See the [evaluation framework](/brightagent/evaluation) for how agent quality is measured, or explore [integrations](/brightagent/integrations) to see what BrightAgent connects to.
# Analysis Agent
Source: https://docs.brighthive.io/brightagent/brightagent_workflows/analysis
The Analyst Agent queries your data, runs statistical analysis, and generates insights — all from natural language questions.
## Overview
The Analyst Agent lets you explore your data through conversation. Ask a question in plain English, and it executes code against your data in a secure **AWS Bedrock sandbox**, performs statistical analysis, and delivers insights — often accompanied by interactive visualizations you can explore further.
## What You Can Ask
* *"What are our top 10 customers by revenue this quarter?"*
* *"Find correlation between marketing spend and revenue growth"*
* *"Identify seasonal trends in sales data over the past 2 years"*
* *"Segment customers based on purchasing behavior"*
* *"Compare this month's performance to last month"*
* *"Detect anomalies in transaction patterns"*
## How It Works
```mermaid theme={null}
graph TD
A[Your Question] --> B[BrightAgent]
B -->|Delegates task| C[Analyst Agent]
C --> D[Reads Data Assets from S3]
D --> E[Executes Code in Bedrock Sandbox]
E --> F{Satisfied with Results?}
F -->|No| E
F -->|Yes| G[Generate Visualization]
G --> H[Return Insights + Chart]
```
1. **BrightAgent delegates** — The orchestrator identifies an analysis task and passes your question along with discovered data assets to the Analyst Agent.
2. **Reads your data** — The agent accesses data files from your organization's S3 storage, loading them into a secure analysis environment.
3. **Executes analysis** — Runs Python code in an **AWS Bedrock Code Interpreter** — a fully isolated sandbox with no access beyond your data.
4. **Iterates until satisfied** — The agent evaluates its own results and refines the analysis if needed, typically completing in 3–5 iterations.
5. **Generates visualizations** — When a chart would help, it creates an interactive **Vega-Lite** visualization rendered directly in BrightAgent.
6. **Delivers insights** — Returns a clear summary with key findings, supporting data, and any charts produced.
## Capabilities
All analysis runs in an **AWS Bedrock sandbox** — fully isolated, stateless, and time-limited. Your data never leaves the secure environment.
Descriptive statistics, distributions, hypothesis testing, correlation analysis, and anomaly detection — all from natural language.
Identifies patterns, seasonality, and changes over time. Compares periods, segments data, and highlights what's changing.
Auto-generates **Vega-Lite** charts with tooltips, responsive layouts, and intelligent chart type selection based on your data.
## Visualization Pipeline
When your question calls for a chart, the Analyst Agent coordinates with a two-phase visualization process:
```mermaid theme={null}
graph TD
A[Analysis Results] --> B[Data Characteristics Analyzed]
B --> C["Phase 1: Plan — Select chart type + columns"]
C --> D["Phase 2: Generate — Create Vega-Lite spec"]
D --> E[Validate Against Schema]
E -->|Invalid| D
E -->|Valid| F[Interactive Chart Rendered]
```
The agent automatically selects the best chart type based on your data:
| Data Pattern | Chart Type |
| --------------------- | --------------------- |
| Categories + numbers | Bar chart |
| Time series | Line or area chart |
| Two numeric variables | Scatter plot |
| Part-of-whole | Arc / pie chart |
| Distributions | Histogram or box plot |
| Correlation matrices | Heatmap |
Charts include tooltips, responsive layouts, and proper formatting — rendered directly in the BrightAgent interface using a **ChartViewer** component.
## Data Access
The Analyst Agent works with data that's already been discovered and prepared:
* **S3 files** — Downloads data from your organization's S3 storage using secure, scoped access
* **Retrieval Agent output** — Receives query results that the Retrieval Agent has already fetched from your warehouse
* **Filesystem** — Reads and writes files in a thread-scoped virtual filesystem backed by S3
The agent does **not** query your warehouse directly — that's the Retrieval Agent's job. The Analyst focuses purely on analysis and insight generation.
## Works With Other Agents
The Analyst Agent frequently collaborates with other agents:
* **Retrieval Agent** finds the right data assets and executes SQL queries before analysis begins.
* **Visualization Agent** handles complex or standalone chart requests.
* **Governance Agent** ensures data access follows your organization's policies.
The Analyst Agent is part of the [BrightAgent architecture](/brightagent/architecture). See [capabilities](/brightagent/capabilities) for the full list of what BrightAgent can do.
# Engineering Agent
Source: https://docs.brighthive.io/brightagent/brightagent_workflows/engineering
The DBT Agent generates data transformation models and submits them as GitHub PRs for your review.
## Overview
The Engineering Agent (DBT Agent) builds data transformation pipelines by generating dbt models from your natural language descriptions. It profiles your raw data, creates properly structured SQL transformations, validates them against your warehouse, and submits everything as a GitHub pull request for your review — nothing gets deployed without your approval.
## Demo: Engineering Agent in Action
*This demo starts at 16:00 and shows the engineering agent building data pipelines and transforming data.*
## What You Can Ask
* *"Create a customer segmentation model based on purchase frequency and total spend"*
* *"Build a transformation that joins orders with customer demographics"*
* *"Generate a dbt model for monthly revenue aggregation by region"*
* *"Transform the raw events table into a session-level summary"*
* *"Create a model that calculates customer lifetime value"*
* *"Build a staging model that cleans and deduplicates the leads data"*
## How It Works
```mermaid theme={null}
graph TD
A[Describe What You Need] --> B[Data Profiling + Schema Detection]
B --> C[Draft SQL Generation]
C --> D{Code Review}
D -->|Needs Changes| E[Code Editing]
E --> D
D -->|Approved| F[Convert to dbt Model]
F --> G[Validate Against Warehouse]
G -->|Failed| H[Review Errors + Fix]
H --> G
G -->|Passed| I[GitHub PR Created]
I --> J[You Review & Approve]
```
1. **Describe what you need** — Tell the agent what transformation you want in plain English.
2. **Agent profiles your data** — Loads table schemas, sample rows, and available reference models to understand your data landscape.
3. **Generates SQL** — Creates an optimized SQL transformation based on your source data structure and requirements.
4. **Code review loop** — An automated review validates the SQL against your schema, data types, and query requirements. If issues are found, the agent edits and resubmits (up to 2 iterations).
5. **Converts to dbt** — Validated SQL is transformed into a properly structured dbt model with `{{ source() }}` references and configurations.
6. **Validates against your warehouse** — The model executes against your actual warehouse to confirm it produces correct results. If validation fails, the agent reviews errors and automatically retries with a built-in retry strategy.
7. **Submits a GitHub PR** — All generated code is submitted as a pull request for your team to review before deployment.
## Key Capabilities
Generates SQL using actual column names, data types, and table structures from your warehouse — not guesses. References existing dbt sources and models from your GitHub repo.
Every generated query goes through automated review that checks schema compatibility, data type alignment, and correctness before proceeding.
Models are executed against your warehouse to verify they produce correct results. Failed validations trigger automatic error analysis and fixes.
Every generated model is submitted as a GitHub PR with full SQL, configurations, and a business-friendly summary — nothing touches your warehouse without approval.
## Multi-Step Pipeline
The Engineering Agent uses an 8-step pipeline that ensures quality at every stage:
| Step | What Happens |
| ------------------ | ----------------------------------------------------------------------------------------------- |
| **Data Profiling** | Loads table schema, sample rows (10 rows), and available reference models from your GitHub repo |
| **SQL Drafting** | Generates optimized SQL based on your request and source data structure |
| **Code Review** | Validates SQL correctness against schema, data types, and query requirements |
| **Code Editing** | Refines SQL based on review feedback — applies fixes, ensures compatibility |
| **dbt Conversion** | Transforms SQL into proper dbt model syntax with `{{ source() }}` references |
| **dbt Validation** | Executes the model against your warehouse and verifies output |
| **Error Review** | If validation fails, analyzes root cause and recommends specific fixes |
| **Finalization** | Packages the model with a summary and submits as a GitHub pull request |
## GitHub Integration
The agent integrates with your GitHub repositories to:
* **Read existing sources** — Fetches `sources.yml` from your dbt project to understand available data sources and reference models
* **Browse repo structure** — Navigates branches and directories to understand your project layout
* **Submit pull requests** — Creates PRs with generated dbt models, configurations, and descriptions for your team to review
## Works With Other Agents
* **Retrieval Agent** provides raw data context, schema information, and identifies which data assets to transform.
* **Analyst Agent** receives clean, transformed data for analysis after models are deployed.
* **Governance Agent** tracks transformation lineage in Neo4j and ensures compliance with data policies.
The Engineering Agent is part of the [BrightAgent architecture](/brightagent/architecture). See [capabilities](/brightagent/capabilities) for the full list of what BrightAgent can do.
# Governance Agent
Source: https://docs.brighthive.io/brightagent/brightagent_workflows/governance
The Governance Agent tracks data quality, manages metadata, and maintains lineage across your data estate.
## Overview
The Governance Agent ensures your data is trustworthy, well-documented, and traceable. It connects to **OpenMetadata** for catalog operations, uses **Neo4j** for lineage tracking, and runs **data quality validations** — giving you confidence that the data behind your decisions is accurate and compliant.
## What You Can Ask
* *"Show me the lineage for the revenue table"*
* *"What tags are applied to our customer data?"*
* *"List all databases in our workspace"*
* *"Run a quality check on the orders dataset"*
* *"Generate a description for the marketing\_leads table"*
* *"What glossary terms do we have for financial data?"*
## How It Works
```mermaid theme={null}
graph TD
A[Your Request] --> B[BrightAgent]
B --> C[Governance Agent]
C --> D{Request Type}
D -->|Metadata & Catalog| E[OpenMetadata MCP Tools]
D -->|Data Quality| F[Quality Validation Workflow]
D -->|Lineage| G[Neo4j Graph Queries]
D -->|Descriptions| H[Description Generation]
E --> I[Results Returned]
F --> I
G --> I
H --> I
```
1. **You ask a governance question** — Anything about data quality, metadata, lineage, tags, glossaries, or catalog operations.
2. **Intent is classified** — The agent determines whether you need metadata lookup, quality validation, lineage tracing, or description generation.
3. **Right tools are invoked** — OpenMetadata MCP tools for catalog operations, Great Expectations for quality checks, or Neo4j for lineage queries.
4. **Results are delivered** — Quality reports, lineage paths, metadata details, or updated descriptions are returned in a clear format.
## Key Capabilities
Runs automated quality checks using **Great Expectations** — completeness, accuracy, consistency, and freshness — with detailed per-column reports.
Traces data from source through transformation to consumption via **Neo4j** graph queries. Know exactly where every data point comes from and what depends on it.
Connects to **OpenMetadata** for full catalog operations — browse databases, search entities, explore schemas, manage tags, glossaries, and classifications.
Automatically generates business-friendly descriptions for data assets using AI analysis of metadata and sample data — then updates your catalog.
## Catalog Operations
The Governance Agent connects to **OpenMetadata** via MCP (Model Context Protocol) for comprehensive catalog operations:
| Category | What You Can Do |
| -------------------------- | ----------------------------------------------------------------- |
| **Tables & Schemas** | Browse tables, schemas, and databases across your workspace |
| **Search & Discovery** | Search entities semantically, get suggestions for matching assets |
| **Tags & Classifications** | List and explore tags, classifications, and their assignments |
| **Glossaries** | Browse business glossaries and terms for shared vocabulary |
| **Test Cases** | View existing data quality test cases and test suites |
| **Lineage** | Trace upstream and downstream dependencies for any entity |
## Data Lifecycle
```mermaid theme={null}
graph TD
A[Data Created / Uploaded] --> B[Auto-Cataloged in Neo4j]
B --> C[Schema Discovered by Glue]
C --> D[Registered in OpenMetadata]
D --> E[Quality Assessment]
E --> F[Description Generated]
F --> G[Tags & Classifications Applied]
G --> H[Available for Analysis]
H --> I[Lineage Tracked End-to-End]
```
## How Lineage Works
Neo4j tracks relationships between every entity in your data estate:
* **Data Assets** — What tables, files, and datasets exist across all sources
* **Organizations** — Which organization provided the data
* **Workspaces** — Which workspaces consume and analyze the data
* **Transformations** — How data was transformed (dbt models, Glue crawlers, SQL queries)
* **Users** — Who accessed and modified what, and when
When you ask *"Where does this metric come from?"*, the Governance Agent traverses the Neo4j graph to show the complete lineage path from raw source to final report.
## Human-in-the-Loop
Quality validations include a **human-in-the-loop** step:
1. The agent analyzes your dataset and generates recommended quality expectations
2. You're presented with the list of proposed checks and can **select which ones to run**
3. Only your approved expectations are executed against the data
4. Results are returned with per-column pass/fail status and detailed statistics
This ensures you stay in control of what gets validated and how quality is measured.
## Works With Other Agents
* **Retrieval Agent** — Governance policies determine what data is accessible and discoverable.
* **Engineering Agent** — Transformation lineage is tracked when dbt models are deployed.
* **Analyst Agent** — Quality scores inform confidence in analysis results.
* **Metadata Agent** — Collaborates on catalog maintenance, schema exploration, and data documentation.
* **Quality Agent** — Runs detailed quality validations as part of the governance workflow.
The Governance Agent is part of the [BrightAgent architecture](/brightagent/architecture). See [capabilities](/brightagent/capabilities) for the full list of what BrightAgent can do.
# Metadata Agent
Source: https://docs.brighthive.io/brightagent/brightagent_workflows/metadata
The Metadata Agent uses OpenMetadata to generate descriptions, understand schemas, enrich your data catalog, and track lineage — keeping your data assets documented and discoverable.
## Overview
The Metadata Agent is responsible for keeping your data catalog current, understandable, and well-documented. It connects to **OpenMetadata (OMD)** via MCP to read and enrich metadata across your entire data estate — generating business-friendly descriptions, managing tags, understanding schema definitions, and tracing data lineage.
## What You Can Ask
* *"Describe the customers table"* — Generates a business-focused description based on schema and sample data
* *"What columns are in the orders dataset?"* — Returns full schema with data types and existing documentation
* *"Add a description to the revenue column"* — Updates column-level metadata in OpenMetadata
* *"Tag the email column as PII"* — Applies sensitivity classifications to columns
* *"Show me the lineage for the sales\_summary table"* — Traces upstream sources and downstream dependencies
* *"What tables are related to inventory?"* — Semantic search across your catalog to find relevant assets
## How It Works
```mermaid theme={null}
graph TD
A[User Request] --> B[Metadata Agent]
B --> C{What's needed?}
C -->|Discovery| D["Vector Search + OpenMetadata"]
C -->|Schema Details| E["OpenMetadata: get_table"]
C -->|Description Generation| F["LLM Analyzes Schema + Sample Data"]
C -->|Metadata Update| G["OpenMetadata: patch_entity"]
C -->|Lineage| H["OpenMetadata: get_entity_lineage"]
D --> I[Results to User]
E --> I
F --> G
G --> I
H --> I
```
## OpenMetadata Integration
The Metadata Agent connects to OpenMetadata through the **Model Context Protocol (MCP)**, providing structured, validated access to your full data catalog.
Browse databases, schemas, and tables. View column names, data types, constraints, and existing documentation — all through the OpenMetadata catalog.
LLM-powered description generation that analyzes schema structure, column patterns, and sample data to produce business-focused descriptions that explain what the data *means*, not just what it contains.
Update table and column descriptions, apply PII sensitivity tags, and add documentation — all written back to OpenMetadata via JSON Patch operations.
Trace data relationships and dependencies across your estate — see which tables feed into which, understand upstream sources, and follow transformations through the pipeline.
## Description Generation
When you ask the Metadata Agent to describe a data asset, it goes beyond reading existing documentation. It uses an LLM to generate business-focused descriptions by:
1. **Retrieving metadata** — Fetches the full schema from OpenMetadata (columns, types, constraints)
2. **Assessing context** — Determines whether the schema alone provides enough context, or if sample data is needed
3. **Fetching sample data** (when needed) — Queries your Redshift warehouse for a representative sample to understand actual data patterns
4. **Generating descriptions** — Produces 2-3 sentence descriptions focused on the distinctive business characteristics of the data — what it represents, how it's used, and what makes it unique
5. **Saving to catalog** — Writes the generated description back to OpenMetadata and Neo4j so it's available across the platform
Descriptions are written for business users — they explain what the data *means* in context, not how it was collected or stored.
## Schema Operations
The Metadata Agent can explore your full OpenMetadata catalog hierarchy:
| Level | Operations |
| ------------ | ----------------------------------------------------------------------- |
| **Database** | List databases, view database details, browse schemas within a database |
| **Schema** | List schemas, view schema details, browse tables within a schema |
| **Table** | List tables, get full table metadata, view column definitions and types |
| **Column** | View data types, constraints, existing descriptions, sensitivity tags |
| **Lineage** | Trace upstream sources and downstream consumers for any table |
## Metadata Enrichment
The agent can update metadata directly in OpenMetadata using structured patch operations:
Add or update table-level descriptions that explain the business purpose and context of each data asset.
Document individual columns with business-friendly explanations — what each field represents and how it should be interpreted.
Apply sensitivity classifications to columns containing personal information — emails, SSNs, phone numbers — using OpenMetadata's PII tagging system.
Semantic search across your entire catalog using vector embeddings — find tables by what they contain, not just what they're named.
## Data Asset Discovery
Finding the right data asset is the first step in any metadata operation. The Metadata Agent uses **confidence-based discovery** to surface the most relevant assets:
| Confidence | Threshold | Behavior |
| ------------------ | ----------------- | ------------------------------------------ |
| **Strong Match** | > 60% similarity | Proceeds automatically with the best match |
| **Possible Match** | 40–60% similarity | Presents options and asks you to confirm |
| **Uncertain** | \< 40% similarity | Asks you to clarify or refine your request |
Discovery searches across both **vector embeddings** (semantic meaning) and **OpenMetadata catalog** (structured metadata) to find assets that match your intent — even if you don't know the exact table name.
## Dual Catalog Architecture
The Metadata Agent works across two complementary systems:
```mermaid theme={null}
graph LR
A[Metadata Agent] --> B["OpenMetadata (Catalog)"]
A --> C["Neo4j (Graph)"]
B --> D["Table schemas, column definitions, descriptions, tags, lineage"]
C --> E["Workspace relationships, access control, data asset graph, GraphRAG"]
```
* **OpenMetadata** stores the detailed catalog metadata — schemas, column definitions, descriptions, PII tags, lineage, and quality metrics
* **Neo4j** stores the relationship graph — how data assets connect to workspaces, organizations, and each other, enabling GraphRAG-powered discovery
When the agent updates a description, it writes to both systems — keeping the catalog and the knowledge graph in sync.
The Metadata Agent works alongside the [Governance Agent](/brightagent/brightagent_workflows/governance) for policy compliance and the [Quality Agent](/brightagent/brightagent_workflows/quality) for data quality checks. Together they keep your data estate documented, governed, and healthy.
# Quality Agent
Source: https://docs.brighthive.io/brightagent/brightagent_workflows/quality
The Quality Agent validates data completeness, accuracy, and consistency — surfacing issues before they reach your reports.
## Overview
The Quality Agent runs automated data quality checks across your data estate using **Great Expectations**. It profiles your data, generates intelligent quality expectations based on column types and distributions, and executes validations — surfacing issues proactively so problems are caught before they reach your reports and dashboards.
## What You Can Ask
* *"Check the quality of our customer table"*
* *"Are there any null values in the orders dataset?"*
* *"Run a freshness check on all our staging tables"*
* *"What's the data quality score for our sales data?"*
* *"Find duplicate records in the user profiles table"*
* *"Validate the web analytics dataset"*
## How It Works
```mermaid theme={null}
graph TD
A[Quality Check Request] --> B[Quality Agent]
B --> C[Step 1: Analyze Dataset Structure]
C --> D[Step 2: Generate Quality Expectations]
D --> E{You Select Which Checks to Run}
E --> F[Step 3: Run Quality Validation]
F --> G[Quality Report with Scores]
```
1. **Analyze dataset structure** — The agent queries your warehouse for a data sample (up to 5,000 rows) and creates a comprehensive profile — column types, null percentages, unique counts, value distributions, and statistical summaries.
2. **Generate quality expectations** — Based on the data profile, an LLM generates a tailored set of quality checks — typically 10–25 high-confidence expectations covering completeness, accuracy, consistency, and more.
3. **You select which checks to run** — The proposed expectations are presented for your review. You choose which ones to execute — keeping you in control of how quality is measured.
4. **Run quality validation** — Selected expectations are executed against your data using Great Expectations. Results include per-column pass/fail status, detailed statistics, and an overall quality score.
## Quality Dimensions
Are required fields populated? What percentage of values are missing or null? Checks null rates against configurable thresholds.
Do values match expected patterns, ranges, and business rules? Validates formats (emails, phones), statistical bounds, and value domains.
Is the data coherent across columns and tables? Checks referential integrity, cross-column relationships, and value set membership.
How recently was the data updated? Is it within expected SLA windows? Monitors timestamps and update frequency.
## Supported Quality Checks
The Quality Agent supports over 50 types of data quality expectations, organized by category:
| Category | Example Checks |
| -------------------- | ------------------------------------------------------------- |
| **Completeness** | Null value detection, non-null proportion thresholds |
| **Uniqueness** | Duplicate detection, compound column uniqueness |
| **Value Ranges** | Min/max bounds, mean/median ranges, z-score outlier detection |
| **Pattern Matching** | Regex validation for emails, phones, and custom formats |
| **Set Membership** | Value-in-set checks, distinct value set validation |
| **Schema** | Column count, column ordering, column type verification |
| **Cross-Column** | Pair comparisons (A > B), multi-column sum validation |
| **Distribution** | Statistical bounds on mean, median, standard deviation |
Each expectation includes a **severity level** (High / Medium / Low) and a **confidence score** so you can prioritize what matters most.
## Quality Reports
Every validation produces a detailed quality report:
* **Dataset Overview** — Total rows, columns, and overall missing value percentage
* **Column Analysis** — Per-column breakdown showing type, null rate, unique count, and quality status
* **Failed Expectations** — Specific details on what failed and why, with counts and examples
* **Overall Quality Score** — Percentage of checks passed, stored as a trackable metric
Quality status per column is displayed as:
* **Pass** — All checks for the column passed
* **Warning** — Some checks passed, some failed
* **Fail** — Critical checks failed for the column
## Human-in-the-Loop
The Quality Agent includes a **human-in-the-loop** step before executing validations:
1. The agent profiles your data and generates recommended expectations
2. You review the proposed checks — each with severity, confidence, and description
3. You select which expectations to run
4. Only approved checks are executed
This prevents unnecessary validations and ensures quality measurement aligns with your business priorities.
## Data Connections
The Quality Agent accesses data through the platform's secure infrastructure:
Queries your dedicated Redshift cluster for data samples via cross-account IAM roles. Supports fully qualified table names with schema isolation.
Reads asset metadata to identify tables, resolve names, and record quality execution results back to the catalog.
Quality reports are stored as formatted documents in your organization's S3 storage for long-term access and auditing.
Validations execute in an ephemeral environment — no persistent configuration required. Each run is isolated and stateless.
## Works With Other Agents
* **Governance Agent** — Quality checks are part of the governance workflow. Quality scores feed into compliance reporting and data lifecycle tracking.
* **Analyst Agent** — Quality issues are flagged before analysis begins so you know your confidence level in the underlying data.
* **Engineering Agent** — Quality checks validate transformation outputs after dbt models run, ensuring transformations produce correct results.
* **Metadata Agent** — Quality scores and execution history are stored as metadata on data assets in your catalog.
The Quality Agent is part of the [BrightAgent architecture](/brightagent/architecture). See [capabilities](/brightagent/capabilities) for the full list of what BrightAgent can do.
# Retrieval Agent
Source: https://docs.brighthive.io/brightagent/brightagent_workflows/retrieval
The Retrieval Agent finds and fetches data from across your data stack — so you never need to know where things live.
## Overview
The Retrieval Agent serves as the data discovery and query layer for BrightAgent. It finds relevant data assets using **vector search** across your metadata catalog, generates optimized SQL, and executes queries against your warehouse — so you can ask for data without knowing which table, schema, or source it lives in.
## Demo: Retrieval Agent in Action
*This demo starts at 10:00 and shows data source connection, extraction, and preparation.*
## What You Can Ask
* *"Show me customer demographics"*
* *"What tables do we have for sales data?"*
* *"Find the students dataset from my warehouse"*
* *"Retrieve the CRM dataset"*
* *"How many orders were placed last quarter?"*
* *"Get revenue by region for Q4"*
## How It Works
```mermaid theme={null}
graph TD
A[Your Question] --> B[BrightAgent]
B -->|"Discover assets"| C[Vector Search + Metadata Catalog]
C --> D{Confidence Assessment}
D -->|"Strong Match > 60%"| E[Retrieval Agent]
D -->|"Possible Match 40-60%"| F[Ask You to Confirm]
D -->|"Uncertain < 40%"| G[Ask You to Clarify]
F --> E
E --> H[Generate SQL]
H --> I[Execute Against Warehouse]
I --> J[Results + Artifact Created]
```
1. **You ask a question** — Any question that references data, whether you know the exact table name or not.
2. **Discovers data assets** — The BrightAgent runs a **vector search** across your metadata catalog to find semantically matching data assets by name, description, and schema.
3. **Assesses confidence** — Each match is scored by similarity. Strong matches proceed automatically; uncertain matches ask for your confirmation.
4. **Generates SQL** — The Retrieval Agent creates an optimized SQL query using your data asset's schema, columns, and any workspace policies.
5. **Executes the query** — Runs the SQL against your **Redshift Serverless** warehouse (or Snowflake, if configured) and returns the results.
6. **Creates an artifact** — Saves the full dataset, metadata, and query details as a reusable artifact that other agents can reference.
## Confidence-Based Discovery
The Retrieval Agent uses **similarity scoring** to ensure it finds the right data before querying. This prevents bad queries and wasted compute:
| Confidence | Threshold | What Happens |
| ------------------ | ----------------- | ------------------------------------------------------------------------------------ |
| **Strong Match** | > 60% similarity | Proceeds directly to SQL generation — the agent is confident it found the right data |
| **Possible Match** | 40–60% similarity | Presents options and asks you to confirm which dataset you mean |
| **Uncertain** | \< 40% similarity | Asks you to clarify or refine your request before proceeding |
Discovery searches across **vector embeddings** (semantic meaning of names and descriptions) and your **platform metadata catalog** (structured metadata, schemas, and relationships) to find assets that match your intent — even if you don't know the exact table name.
## SQL Generation & Execution
Once the right data asset is identified, the Retrieval Agent handles the full query lifecycle:
Generates SQL using the actual column names, data types, and table structure from your metadata catalog — not guesses.
Respects **workspace policies** during query generation. If your workspace has data access restrictions, the SQL enforces them.
Queries execute via **cross-account IAM** roles against your dedicated Redshift Serverless cluster. Results are capped at 10,000 rows by default.
Every query result is saved as an **artifact** with full metadata — SQL used, tables queried, column definitions, and a searchable summary.
## What It Connects To
The primary search target — all data asset metadata, schemas, lineage, and relationships live here. Embeddings enable semantic search.
Your workspace data warehouse where analytical queries execute. Auto-scaling, 3-AZ deployment, schema-per-organization isolation.
Organization-level raw data storage. Redshift Spectrum queries S3 data directly without loading it into the warehouse.
Schema metadata auto-discovered by Glue crawlers when new data lands in S3. Feeds into Neo4j for unified search.
## Works With Other Agents
The Retrieval Agent is typically the **first agent invoked** in any data workflow:
* **Analyst Agent** uses retrieved data assets for statistical analysis and exploration.
* **Visualization Agent** receives query results to create interactive charts.
* **Engineering Agent** uses schema context to generate appropriate dbt transformation models.
* **Governance Agent** reports on data lineage and tracks access patterns.
The Retrieval Agent is part of the [BrightAgent architecture](/brightagent/architecture). See the [evaluation framework](/brightagent/evaluation) for how retrieval quality is measured.
# Slack Agent (Beta)
Source: https://docs.brighthive.io/brightagent/brightagent_workflows/slack
The Slack Router Agent brings BrightAgent into Slack — query your data, manage Jira tickets, search Notion, and more without leaving your workspace.
## Overview
The Slack Agent (Slack Router) lets your team interact with BrightAgent directly from Slack. Ask data questions, create Jira tickets, search Notion docs, find Google Drive files, and more — all without leaving your messaging workspace. A fast intent classifier routes each message to the right backend.
## What You Can Ask in Slack
* *"Show me our top customers by revenue"* — Routes to BrightAgent for data queries
* *"Create a Jira ticket for the data pipeline bug"* — Routes to Jira via MCP
* *"Find the doc about our onboarding process"* — Routes to Notion via MCP
* *"Search for the Q4 report in Drive"* — Routes to Google Drive via MCP
* *"What is the difference between a data lake and a warehouse?"* — Routes to general chat
## How It Works
```mermaid theme={null}
graph TD
A[Slack Message] --> B[Intent Classifier]
B --> C{Route by Intent}
C -->|Data Question| D[BrightAgent Platform]
C -->|Jira| E[Jira MCP]
C -->|Notion| F[Notion MCP]
C -->|Google Drive| G[GDrive MCP]
C -->|MS Teams| H[Teams MCP]
C -->|General| I[General Chat]
```
1. **Message arrives from Slack** — User sends a message in a connected Slack channel or DM.
2. **Intent classification** — A fast LLM classifier determines what the user needs and routes to the right backend.
3. **Backend processes the request** — Data questions go to the BrightAgent. Tool requests go to MCP-powered integrations. General questions get a direct chat response.
4. **Response sent back to Slack** — Results are formatted for Slack and posted as a reply.
## Supported Integrations
Full data platform access — query data, run analysis, generate visualizations, check data quality — all from Slack.
Create tickets, check sprint status, view your issues, and manage project tracking via MCP.
Search pages, create documents, and browse your team's wiki and knowledge base via MCP.
Find files, list folders, and access shared documents via MCP.
## Architecture
* **Intent Classifier** — Uses Claude Haiku for fast (\~130ms) intent classification.
* **MCP Tool Execution** — Uses the ReAct pattern for multi-step tool calls against external services.
* **General Chat** — Powered by Amazon Bedrock Nova for quick conversational responses.
* **Workspace Mapping** — Each Slack workspace maps to a Brighthive workspace for proper authentication and data isolation.
# Visualization Agent
Source: https://docs.brighthive.io/brightagent/brightagent_workflows/visualisation
The Visualization Agent creates interactive charts and dashboards from your data — just describe what you want to see.
## Overview
The Visualization Agent transforms your data into interactive charts and visualizations. Describe what you want to see in plain English, and it analyzes your data, selects the right chart type, generates a validated **Vega-Lite** specification, and renders it directly in the BrightAgent interface — with tooltips, responsive layouts, and proper formatting.
## Demo: Visualization Agent in Action
*This demo starts at 14:15 and shows the visualization agent creating charts and dashboards.*
## What You Can Ask
* *"Show me a bar chart of sales by region"*
* *"Create a trend line of customer acquisition costs over the past year"*
* *"Display the correlation between marketing spend and revenue"*
* *"Generate a breakdown of our top product categories"*
* *"Compare this quarter's metrics to last quarter as a pie chart"*
* *"Show me a heatmap of activity by day and hour"*
## How It Works
```mermaid theme={null}
graph TD
A[Your Request] --> B[Data Retrieved or Prepared]
B --> C["Phase 1: Analyze Data + Plan Chart"]
C --> D["Phase 2: Generate Vega-Lite Spec"]
D --> E{Schema Valid?}
E -->|No| D
E -->|Yes| F[ChartViewer Rendered in BrightAgent]
```
1. **You describe what you want** — Natural language request for any visualization, with optional chart type preferences.
2. **Data is prepared** — The Analyst or Retrieval Agent queries your warehouse and returns results, or the agent reads from existing artifacts.
3. **Phase 1: Planning** — An LLM analyzes a data sample to understand column types, distributions, and patterns. It selects the optimal chart type and maps columns to visual encodings.
4. **Phase 2: Generation** — A more capable LLM produces a complete Vega-Lite v6 specification following the plan — including axes, colors, tooltips, and responsive layout.
5. **Validation** — The specification is validated against the Vega-Lite schema. If invalid, the agent applies a retry strategy with error context until the specification passes.
6. **Rendered in BrightAgent** — The chart appears as an interactive **ChartViewer** component with hover tooltips, responsive scaling, and proper formatting.
## Chart Type Selection
The agent automatically recommends the best visualization based on your data characteristics:
| Data Pattern | Chart Type | When Used |
| --------------------- | ------------------------ | ----------------------------------------- |
| Categories + numbers | **Bar chart** | Comparing values across groups |
| Time series | **Line chart** | Showing trends over time |
| Filled time series | **Area chart** | Emphasizing volume or cumulative trends |
| Two numeric variables | **Scatter plot** | Exploring relationships and correlations |
| Part-of-whole | **Arc / donut** | Showing composition or proportions |
| Distributions | **Histogram / box plot** | Understanding data spread and outliers |
| Dense matrices | **Heatmap** | Showing patterns across two dimensions |
| Geographic data | **Map-based** | Spatial analysis and regional comparisons |
You can always override the recommendation — just specify the chart type you want.
## Interactive Features
Every chart includes built-in interactivity:
Hover over any data point to see detailed values. Tooltips automatically include relevant fields from your data.
Charts automatically scale to fit the available space — from full-width dashboards to mobile views.
Categories are automatically distinguished by color using accessible palettes. Numeric values can use gradient scales.
Numbers, dates, currencies, and percentages are formatted appropriately based on data type — no manual configuration needed.
## Data Sources
The Visualization Agent receives data through multiple pathways:
* **From the Analyst Agent** — Pre-analyzed data ready for charting, often as part of a multi-step workflow
* **From the Retrieval Agent** — Raw query results from your warehouse, passed as artifacts with full metadata
* **From S3 files** — Data files generated by other agents or uploaded directly, read from your organization's storage
The agent handles data loading, type detection, and any necessary transformations before generating the chart.
## Works With Other Agents
* **Analyst Agent** provides aggregated query results and statistical insights that feed directly into visualizations.
* **Retrieval Agent** identifies and fetches the right data assets, creating artifacts the Visualization Agent can consume.
* **BrightAgent** coordinates multi-step workflows like *"analyze sales by region and chart it"* — routing through Retrieval, Analyst, and Visualization in sequence.
The Visualization Agent is part of the [BrightAgent architecture](/brightagent/architecture). See [capabilities](/brightagent/capabilities) for the full list of what BrightAgent can do.
# BrightAgent Capabilities
Source: https://docs.brighthive.io/brightagent/capabilities
BrightAgent is your data team in a box — specialized AI agents that handle retrieval, analysis, engineering, visualization, governance, and more.
![BrightAgent Capabilities]()
## One Interface, Many Agents
You interact with one AI assistant. Behind the scenes, a **BrightAgent** orchestrates specialized agents to handle your request — analyzing intent, routing to the right experts, coordinating multi-step workflows, and synthesizing results into a clear response.
The BrightAgent doesn't just pick one agent per query. A question like *"show me a chart of sales by region"* triggers three agents in coordination: the Retrieval Agent finds the right data, the Analyst Agent queries and aggregates it, and the Visualization Agent produces the chart. You see one seamless answer.
## Specialized Agents
Finds and fetches data from your warehouse, data lake, and metadata catalog so you don't have to know where things live.
Queries your data, runs statistical analysis, and generates Jupyter notebooks with insights — all from a natural language question.
Generates dbt models for data transformation and submits them as GitHub PRs for your review before deployment.
Creates interactive charts and visualizations from your data — just describe what you want to see.
Tracks data quality, manages metadata, and maintains lineage across your data estate via Neo4j.
Validates data completeness, accuracy, and consistency — surfacing issues before they reach your reports.
Uses OpenMetadata to generate descriptions, understand schemas, enrich your catalog with tags and documentation, and track lineage.
Interact with BrightAgent directly from Slack — query data, manage Jira tickets, search Notion, and more.
## How Agents Collaborate
BrightAgent isn't a collection of isolated tools — agents coordinate to handle complex, multi-step tasks that span the entire data lifecycle.
### Multi-Agent Workflow Example
```mermaid theme={null}
graph TD
A["'Analyze customer churn and show me a chart'"] --> B[BrightAgent]
B --> C[Retrieval Agent]
C --> D["Searches Neo4j for customer data assets"]
D --> E[Analyst Agent]
E --> F["Generates & executes SQL in Redshift"]
F --> G[Visualization Agent]
G --> H["Produces interactive chart"]
H --> I[BrightAgent Synthesizes Response]
I --> J["Natural language summary + chart + data"]
```
### Coordination Patterns
Multiple agents work simultaneously when tasks are independent — data retrieval and visualization setup can run in parallel to reduce response time.
Some workflows require step-by-step execution: Retrieval first finds data, then Analysis queries it, then Visualization charts the results.
Agents share relevant context and intermediate results through shared state — the Analyst Agent knows exactly which data the Retrieval Agent found.
## What You Can Ask
BrightAgent handles the full range of data operations through natural language:
| What You Need | What Happens |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------- |
| *"Find the customer dataset"* | Retrieval Agent searches Neo4j metadata, discovers matching data assets, presents options |
| *"Analyze sales trends by quarter"* | Retrieval finds data → Analyst generates SQL, executes query, produces statistical insights |
| *"Create a bar chart of revenue by region"* | Retrieval → Analyst → Visualization coordinate to produce an interactive Plotly chart |
| *"Build a dbt model for customer segmentation"* | Engineering Agent generates dbt SQL, configurations, and tests — submitted as a GitHub PR |
| *"Check data quality for the orders table"* | Quality Agent runs completeness, accuracy, consistency, and freshness checks |
| *"Who owns the marketing dataset?"* | Metadata Agent queries Neo4j for ownership, descriptions, and access information |
| *"Show me the lineage for this report"* | Governance Agent traces data lineage through Neo4j — from source to final output |
## Human-in-the-Loop
BrightAgent is designed so that AI assists your workflow without making irreversible changes autonomously:
* **dbt Models** — Generated transformation code is submitted as a GitHub PR. Your team reviews before merging.
* **Code Generation** — Jupyter notebooks and analysis scripts are presented for review before execution.
* **Governance Actions** — Changes to data policies and access controls require explicit user confirmation.
## Continuous Quality
Every agent interaction is evaluated in real-time for relevance, correctness, and goal accuracy. Quality metrics are tracked across all agents and fed back into continuous improvement.
Learn about the [evaluation framework](/brightagent/evaluation) that keeps BrightAgent reliable, or explore the [architecture](/brightagent/architecture) to understand how agents are built.
# Evaluation & Quality
Source: https://docs.brighthive.io/brightagent/evaluation
Multi-layered evaluation framework ensuring agent accuracy, correctness, and reliability across the entire lifecycle — from development through production.
## Overview
BrightAgent uses a multi-layered evaluation strategy that covers every stage of the agent lifecycle. Pre-flight checks validate infrastructure before agents run. Runtime evaluations score every agent response for relevance and correctness. Post-flight checks verify outputs before they reach users. And SDLC evaluations ensure the platform maintains high standards across every release.
```mermaid theme={null}
graph TD
A[Pre-Flight] --> B[Runtime Evaluation]
B --> C[Post-Flight Verification]
C --> D[Response Delivered]
E[SDLC Evals] --> F[CI/CD Pipeline]
F --> G[PR Approval Gate]
G --> H[Production Deploy]
```
## Pre-Flight Evaluation
Before agents handle user queries, pre-flight checks ensure the underlying infrastructure and services are functioning correctly.
Core platform functions (authentication, API connectivity, data access) are validated with deterministic tests that use custom comparison functions to verify outputs match expected criteria.
Curated test suites define expected inputs and outputs for each agent. Single-turn tests validate one-off queries. Multi-turn tests validate conversational workflows across multiple interactions.
Agent runners are tested against known scenarios before deployment — verifying that the Retrieval Agent finds data, the Analyst Agent generates valid SQL, and the Visualization Agent produces charts.
All agent responses are grounded in actual data from Neo4j and Redshift — not hallucinated from training data. Pre-flight checks verify that context sources are accessible and returning expected schemas.
## Runtime (Online) Evaluation
Every agent interaction is scored in real-time using DeepEval metrics, providing continuous measurement of response quality.
### Single-Turn Metrics
For individual user queries, two metrics are measured:
| Metric | What It Measures | How It Works |
| -------------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| **Answer Relevancy** | Is the response relevant to the user's question? | DeepEval's AnswerRelevancyMetric scores the semantic alignment between input and output |
| **Correctness** | Is the response accurate and logically sound? | GEval (LLM-as-a-judge) evaluates whether the agent successfully executed the request without errors |
The Correctness metric operates in two modes: **strict mode** compares against expected outputs when available, and **open mode** evaluates correctness based on the input alone — catching logical errors even without a reference answer.
### Multi-Turn Metrics
For conversational workflows that span multiple interactions, four additional metrics track quality across the full conversation:
| Metric | What It Measures |
| ----------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Turn Relevancy** | Are responses relevant throughout the entire conversation, not just the first message? |
| **Knowledge Retention** | Does the agent remember information from earlier turns? (e.g., "use the dataset I mentioned earlier") |
| **Conversation Completeness** | Does the conversation achieve its intended goal by the final turn? |
| **Goal Accuracy** | How precisely did the agent accomplish what the user set out to do? |
### LLM-as-a-Judge
The Correctness metric uses GEval — an LLM-as-a-judge approach that evaluates whether agent output is a correct and appropriate response to the input. The judge evaluates:
* Whether the agent successfully executed the user's request
* Whether the output is accurate, complete, and logically sound
* Whether the response matches expected output (when provided)
This catches subtle errors that pattern-matching can't — like SQL that runs without errors but returns the wrong data.
### Tool Validation via MCP
Model Context Protocol (MCP) integration validates that agent tool calls are well-formed and authorized before execution. If an agent tries to call a tool with invalid parameters or access unauthorized resources, MCP blocks the call before it reaches your infrastructure.
## Post-Flight Verification
Before a response reaches the user, multiple verification layers ensure quality.
The BrightAgent can route results through multiple agents for validation. For example, the Governance Agent verifies that an Analyst Agent's query respects data access policies before results are returned.
Operations that modify infrastructure — dbt models, schema changes, governance policies — require explicit human approval. Generated code is submitted as a GitHub PR, not executed automatically.
Every agent interaction is logged with the full chain of tool calls, data accessed, and decisions made. Users can inspect the SQL generated, the data assets queried, and the reasoning behind each response.
Responses include traceability to the data sources used — which tables were queried, which Neo4j metadata informed the response — so users can verify the answer's foundation.
## SDLC Evaluation
Platform-level evaluations run as part of the software development lifecycle to ensure every release maintains quality standards.
### CI/CD Pipeline
Evaluations run automatically on every pull request via GitHub Actions:
```mermaid theme={null}
graph TD
A[Developer Pushes Code] --> B[GitHub Actions Triggered]
B --> C[Install Dependencies]
C --> D[Run Agent Evaluations]
D --> E[Score Against Metrics]
E --> F[Post Results as PR Comment]
F --> G{Scores Pass Threshold?}
G -->|Yes| H[Ready for Review]
G -->|No| I[Fix Required]
```
Evaluation results are posted directly as PR comments, showing per-test-case scores with color-coded badges (green/yellow/red) so reviewers can see quality impact at a glance.
### Parallel Test Execution
The evaluation framework runs test cases concurrently for faster feedback:
* **Single-turn tests**: Up to 10 concurrent executions
* **Multi-turn tests**: Up to 5 concurrent conversations (with sequential turns within each to maintain state)
* **CI mode**: Reduced parallelism to conserve resources in pipeline environments
### Observability
Every evaluation metric is recorded via OpenTelemetry — tracking answer relevancy, correctness, turn relevancy, and goal accuracy across all agents and test types.
Full trace visibility into every agent step — from initial intent classification through tool calls to final response synthesis. Traces include latency breakdowns, token usage, and error attribution.
Summary statistics across all test cases — average scores per metric, pass/fail rates, and trend analysis across releases.
Results are available in console, JSON, and Markdown formats — supporting local debugging, programmatic analysis, and GitHub PR comments.
### What Gets Measured
| Category | Metrics |
| --------------------- | ------------------------------------------------------------------------------------------------------------ |
| **Agent Quality** | Answer relevancy, correctness, turn relevancy, knowledge retention, conversation completeness, goal accuracy |
| **Operational** | Agent invocations, latency (p50/p95/p99), error rates, token usage per model |
| **User Satisfaction** | Helpfulness ratings, accuracy feedback, response speed perception |
| **Infrastructure** | Guardrails blocks (PII detections, content violations), hallucination rate |
Evaluation metrics and quality scores feed directly into continuous improvement — helping the BrightAgent team identify and fix quality regressions before they reach production. For more on how agents work, see the [BrightAgent architecture](/brightagent/architecture).
# Integrations
Source: https://docs.brighthive.io/brightagent/integrations
BrightAgent connects to your entire data stack — warehouse, catalog, transformations, and collaboration tools — through a secure, unified integration layer.
## How BrightAgent Connects
BrightAgent integrates with the services that power your data stack through the **Datapiary** library, providing a uniform interface across all service types. Every interaction is authenticated, scoped to your workspace, and tracked in Neo4j for full lineage and auditability.
```mermaid theme={null}
graph TD
A[BrightAgent] --> B[Datapiary Integration Layer]
B --> C[Warehouse Services]
B --> D[Catalog Services]
B --> E[Transformation Services]
B --> F[Ingestion Services]
B --> G[Collaboration Services]
B --> H[External Tools via MCP]
C --> C1[Redshift Serverless]
C --> C2[Snowflake]
D --> D1[Neo4j]
D --> D2[Glue Data Catalog]
D --> D3[OpenMetadata]
E --> E1[DBT Cloud]
F --> F1[S3 Data Lake]
F --> F2[Airbyte]
G --> G1[Stream.io]
H --> H1[Jira]
H --> H2[Notion]
H --> H3[Google Drive]
```
## Core Integrations
**Metadata & Knowledge Graph** — Single source of truth for all metadata, lineage, relationships, and data asset information. Every agent queries Neo4j for context via GraphRAG. Stores user, workspace, organization relationships; data asset schemas and locations; transformation lineage; and access control metadata.
**Data Warehouse** — Auto-scaling analytical warehouse deployed across 3 availability zones with schema-per-organization isolation. The Analyst Agent generates and executes SQL here. Queries access organization data via cross-account IAM and Redshift Spectrum — reading S3 in place without copying.
**Data Warehouse (via Datapiary)** — Available for organizations that need Snowflake alongside Redshift. Data syncs from organization S3 to Snowflake via cross-account IAM. DBT Cloud transformations can run against either warehouse.
**Schema Discovery** — Glue crawlers automatically detect schemas when data lands in S3 — inferring column names, data types, partitions, and formats. Metadata is synced to Neo4j and made available to all agents immediately.
**Data Transformation (via Datapiary)** — The Engineering Agent generates dbt models that run on DBT Cloud. All generated code goes through GitHub PRs for human review. Neo4j tracks transformation lineage — which models depend on which sources.
**Data Lake Storage** — Each organization gets dedicated S3 buckets (raw, staged, shared) in their own AWS account. File uploads trigger automatic schema discovery via Glue and metadata registration in Neo4j.
**Data Ingestion (Optional)** — Self-hosted Airbyte instance with 300+ connectors for ingesting data from external sources like Shopify, HubSpot, Salesforce, PostgreSQL, and hundreds more. Runs within the organization's dedicated AWS account.
**Metadata Catalog** — Unified metadata catalog integration for comprehensive data asset discovery, documentation, and lineage tracking. Connected via MCP for direct agent access.
## MCP Integrations (Model Context Protocol)
BrightAgent uses MCP for validated access to external tools and services. MCP ensures that every tool call is well-formed, authorized, and auditable before execution.
Create tickets, update statuses, and manage sprints directly from BrightAgent or Slack. The Slack Router Agent routes Jira-related requests to the Jira MCP server.
Search pages, query databases, and retrieve documentation from Notion workspaces. Integrated as an MCP server for structured access.
Search and retrieve documents from Google Drive. Available through the Slack Router Agent for quick access from Slack conversations.
Direct MCP connection to OpenMetadata for metadata discovery, data quality information, and catalog operations beyond what's stored in Neo4j.
## Observability & Tracing
Full distributed tracing for every agent interaction — from initial user query through intent classification, tool calls, and response synthesis. Traces include latency breakdowns per agent, token usage by model, and error attribution.
Evaluation metrics, agent invocation counts, latency percentiles (p50/p95/p99), and error rates are recorded via OpenTelemetry for operational dashboards and alerting.
## Integration Architecture
BrightAgent doesn't connect to data services directly from the AI layer. Instead, all access flows through the platform's secure infrastructure:
```mermaid theme={null}
graph LR
A[BrightAgent] -->|"Authenticated Request"| B[Platform API]
B -->|"JWT Validation"| C[Cognito Auth]
B -->|"Metadata Lookup"| D[Neo4j]
B -->|"Cross-Account IAM"| E[Customer Infrastructure]
E --> F["Redshift (Workspace Account)"]
E --> G["S3 (Organization Account)"]
E --> H["Glue Catalog (Organization Account)"]
```
This architecture means:
* **All access is authenticated** via Cognito JWT tokens — every request is verified before reaching any backend service
* **All queries respect workspace boundaries** — agents can only access data the user's workspace is authorized for
* **All interactions are logged** in Neo4j for lineage and audit — you can trace exactly what data was accessed and why
* **No credentials are shared** — cross-account access uses IAM role assumption (AWS STS), not stored passwords or API keys
## Service Categories
The Datapiary library organizes integrations into service types, providing a consistent interface regardless of the underlying technology:
| Category | Services | What Agents Use Them For |
| ------------------ | -------------------------------------- | -------------------------------------------------------------------- |
| **Warehouse** | Redshift Serverless, Snowflake | Executing SQL queries, running analysis, aggregating data |
| **Catalog** | Neo4j, Glue Data Catalog, OpenMetadata | Discovering data assets, understanding schemas, tracking lineage |
| **Transformation** | DBT Cloud | Generating and running data transformation models |
| **Ingestion** | S3 direct upload, Airbyte | Bringing data into the platform from files and external sources |
| **Notebook** | Jupyter (E2B sandbox) | Generating and executing analysis notebooks in isolated environments |
| **Collaboration** | Stream.io | Real-time team chat and collaboration within the platform |
| **External Tools** | Jira, Notion, Google Drive, MS Teams | Task management, documentation, and file access via MCP |
Learn about the [platform infrastructure](/platform/backend) that powers these integrations, or see the [security model](/platform/security) for how data isolation and access control work.
# Introduction
Source: https://docs.brighthive.io/getting_started/introduction
Welcome to Brighthive - The AI-powered Agentic DataOps platform that gives every team a complete data stack.
## What is Brighthive?
Brighthive is an Agentic DataOps platform that gives every team a complete data stack. Powered by specialized AI agents that work together seamlessly, Brighthive handles ingestion, governance, querying, data engineering, and visualization — equipping everyone in the enterprise with capabilities that previously required a dedicated data team.
## Our Mission
Transform knowledge work to be data-informed work by giving a "data team in a box" to everyone.
## The Problem We Solve
Modern organizations generate vast amounts of data, but most employees struggle to access and use it effectively. The complexity of today's data stacks, combined with siloed teams and specialized tools, means that valuable data assets remain locked away. This is especially acute in middle-market organizations where data is plentiful but resources for dedicated data teams are limited.
Brighthive solves this by providing an AI-powered, end-to-end data platform that democratizes data access — enabling everyone to become their own data analyst and make more informed decisions.
## How It Works
Brighthive uses a workspace and organization model to keep your data secure and well-organized:
* **Organizations** provide data. Each organization gets a dedicated, isolated AWS account with its own S3 data lake and schema catalog.
* **Workspaces** provide centralized analytics. Each workspace gets a dedicated data warehouse (Redshift Serverless) where organization data is queryable.
* **Each gets dedicated, isolated infrastructure** — your data never co-mingles with other customers.
This architecture means organizations contribute data while workspaces provide the analytical horsepower — all connected securely through cross-account access controls.
## Our Approach
A unified platform handling ingestion, transformation, warehousing, and governance
Specialized data agents working together to manage critical data tasks through conversation
Dedicated AWS accounts per customer with isolated infrastructure and security boundaries
Stand up your complete data stack in minutes, not months
## Key Capabilities
Upload files directly to S3 or connect 300+ sources via Airbyte — auto-discovered and cataloged
AI-generated dbt models submitted as GitHub PRs for your review and approval
Neo4j knowledge graph tracking metadata, lineage, and relationships across your entire data estate
Redshift Serverless with auto-scaling across 3 availability zones, plus Snowflake via Datapiary
Upload data files with automatic schema detection and catalog registration
Explore your data catalog, collaborate with your team, and interact with BrightAgent
## Get Started
Sign up for a trial and experience the platform firsthand
Visit our website for more information
## About Brighthive
Brighthive is a venture-backed AI data management and analysis startup headquartered in downtown Chicago, IL. We're committed to democratizing data access and empowering organizations to unlock the full potential of their data assets.
For more information, visit [www.brighthive.io](https://www.brighthive.io) or sign up for a trial to experience our platform in action.
# Platform Overview
Source: https://docs.brighthive.io/getting_started/overview
A high-level overview of the Brighthive platform architecture and its key differentiators.
## High-Level Architecture
Brighthive uses a three-tier architecture that provides dedicated, isolated infrastructure for every customer:
```mermaid theme={null}
%%{init: {"theme": "dark"}}%%
graph TD
User[End Users] --> WebApp[React WebApp]
WebApp --> GraphQLAPI[GraphQL API - Apollo Federation]
WebApp --> BrightAgent[BrightAgent - AI Agents]
GraphQLAPI --> Neo4j[Neo4j - Metadata SSOT]
GraphQLAPI --> Cognito[Cognito Auth]
BrightAgent --> Neo4j
subgraph workspace["Workspace Account"]
Redshift[Redshift Serverless - 3 AZ]
Snowflake[Snowflake via Datapiary]
DBTCloud[DBT Cloud]
end
subgraph org["Organization Account"]
S3[S3 Data Lake]
Glue[Glue Data Catalog]
OrgRole[OrgDataCatalogRole - IAM]
end
GraphQLAPI --> Redshift
BrightAgent --> Redshift
Redshift -->|Cross-account IAM| OrgRole
OrgRole --> S3
OrgRole --> Glue
Snowflake --> OrgRole
DBTCloud --> Redshift
```
**Tier 1: Shared Platform** — The webapp, GraphQL API, BrightAgent AI system, and Neo4j metadata graph run on shared infrastructure.
**Tier 2: Workspace Accounts** — Each customer workspace gets a dedicated AWS account with Redshift Serverless (3-AZ), Snowflake (via Datapiary), and DBT Cloud for transformations.
**Tier 3: Organization Accounts** — Each data-providing organization gets a dedicated AWS account with S3 storage, Glue Data Catalog, and cross-account IAM roles for secure data sharing.
***
## Key Differentiators
* **Dedicated Infrastructure**: Every customer gets isolated AWS accounts — not shared tenancy. Your data warehouse, storage, and compute are yours alone.
* **AI-Native**: BrightAgent's multi-agent system (LangGraph) is built into the platform, not bolted on. Ask questions in plain English and get analysis, visualizations, and dbt models.
* **Graph-Powered Metadata**: Neo4j serves as the single source of truth for all metadata, lineage, and relationships across your entire data estate.
* **Cross-Account Security**: Organization data is shared to workspaces through IAM-based cross-account roles — no data copying, no shared credentials.
* **Automated Schema Discovery**: Glue crawlers auto-detect schemas when data lands in S3, register metadata in Neo4j, and make it immediately queryable.
## Platform Components
Auto-scaling data warehouse across 3 availability zones with schema-per-organization isolation
Single source of truth for all metadata, lineage, data asset relationships, and GraphRAG capabilities
Automatic schema discovery via crawlers when data lands in S3 — no manual cataloging required
Multi-agent orchestration system with specialized agents for retrieval, analysis, visualization, engineering, and governance
Modern webapp with real-time collaboration via Stream.io, data catalog browsing, and BrightAgent interface
Agent-generated dbt models submitted as GitHub PRs, with transformation lineage tracked in Neo4j
## Key Data Flows
### User Query
```
User → WebApp → GraphQL API → Neo4j (metadata lookup) → Workspace Redshift → Org S3/Glue (cross-account) → Results
```
### Data Upload
```
File → Org S3 → EventBridge → Glue Crawler → Schema Discovery → Neo4j Metadata Sync → Available for Querying
```
### AI Assistant
```
User Question → BrightAgent → Specialized Agents (Retrieval, Analyst, Visualization) → Neo4j + Redshift → Response
```
### Provisioning
```
Admin Request → brighthive-admin Step Functions → Create AWS Account → Deploy CDK → Neo4j + DynamoDB → Ready
```
# Backend API
Source: https://docs.brighthive.io/platform/backend
Apollo GraphQL Federation API on AWS Lambda with Neo4j, Cognito auth, and supporting services.
## Overview
The Brighthive backend is a GraphQL API built with Apollo Federation v2, deployed on AWS Lambda behind API Gateway. It serves as the central coordination layer between the webapp, BrightAgent, and all customer infrastructure.
## Architecture
Apollo Federation v2 running on Lambda + API Gateway. Provides a unified query interface for all platform operations at `api.{env}.brighthive.net`.
Graph database on EC2 storing all metadata, lineage, user/workspace/org relationships, and data asset catalog. The single source of truth for the entire platform.
Two user pools — Platform (customer users) and Internal (admin). JWT tokens authenticate all API requests via API Gateway custom authorizers.
Stores account mappings (`S3BucketsByAccount`) and data asset references (`TableIdsByDataAssetUuid`) for fast lookups.
## Key Responsibilities
* **Authentication & Authorization** — Cognito JWT validation, workspace-scoped access control.
* **Data Catalog Operations** — CRUD operations on data assets, schemas, and metadata in Neo4j.
* **Workspace Coordination** — Routes queries to the correct workspace's Redshift API based on Neo4j metadata.
* **User Management** — User creation, workspace membership, role assignment.
* **Ingestion Orchestration** — Coordinates file uploads, Airbyte connections, and data onboarding workflows.
* **Service Integration** — Connects to OpenMetadata, Stream.io, Redis, and customer infrastructure.
## Supporting Services
* **Redis** — Caching layer for frequently accessed metadata and API responses.
* **OpenMetadata** — Unified metadata catalog integration via the Internal API stack.
* **Stream.io** — Powers real-time collaboration and chat within the webapp.
* **S3 + CloudFront** — Static asset hosting and delivery.
## API Endpoints
* `api.{env}.brighthive.net` — Main GraphQL API (Apollo Federation).
* `api.{env}.brighthive.net/ogm` — Neo4j Object-Graph Mapping endpoint.
## How Queries Flow
```
User (webapp) → API Gateway → Cognito Authorizer → GraphQL Lambda → Neo4j (metadata) → Workspace Redshift API → Results
```
The GraphQL Lambda looks up workspace metadata in Neo4j (account ID, Redshift API URL), then calls the workspace's Redshift REST API to execute the query. Results flow back through the same path.
# Graph
Source: https://docs.brighthive.io/platform/data_graph
Neo4j serves as the single source of truth for all metadata, lineage, and relationships across the Brighthive platform.
Neo4j is the backbone of the Brighthive platform — every data asset, user, workspace, organization, and transformation is represented as a connected graph node.
## The Single Source of Truth
Neo4j stores and connects all platform metadata. When BrightAgent searches for data, when the webapp displays your catalog, or when lineage is traced from source to report — it all comes from Neo4j.
### What Neo4j Tracks
* **Users** — Platform users, their roles, and workspace memberships.
* **Workspaces** — Customer workspaces with their AWS account IDs, Redshift API URLs, and configurations.
* **Organizations** — Data-providing organizations, their AWS accounts, and S3 bucket locations.
* **Data Assets** — Every table, file, and dataset with schema, row counts, and location metadata.
* **Lineage** — How data flows from source through transformation to consumption.
* **Relationships** — Which organizations belong to which workspaces, which users can access what.
## Why Neo4j?
Powers Graph Retrieval-Augmented Generation with native vector search and graph traversal — giving BrightAgent rich context for every query.
Cypher query language enables fast, intuitive graph pattern matching — find connections across your data estate in milliseconds.
Native graph structure is ideal for tracking data lineage — from raw source through transformations to final reports.
Combines vector similarity search with graph relationships for semantic data discovery.
## GraphRAG
GraphRAG enhances traditional RAG by leveraging knowledge graphs to provide richer context and more accurate responses. BrightAgent uses GraphRAG to:
* **Find semantically similar data assets** while considering graph relationships.
* **Traverse multiple relationship levels** to gather comprehensive context for complex questions.
* **Link mentions across queries** through graph connections — e.g., understanding that "revenue" might refer to data in `fact_orders` or `sales_summary`.
* **Adapt retrieval strategy** based on data lineage and relationship patterns.
## How the Platform Uses Neo4j
### Query Routing
When a user queries data through the webapp or BrightAgent:
1. GraphQL API queries Neo4j for the workspace's Redshift API URL and account ID.
2. Neo4j returns the metadata needed to route the query to the correct workspace infrastructure.
### Data Catalog
The webapp's data catalog view is powered entirely by Neo4j:
* Browse data assets with schema details, tags, and quality scores.
* See relationships between tables, organizations, and workspaces.
* Trace lineage from raw source to final output.
### Agent Context
BrightAgent's Retrieval Agent queries Neo4j to find relevant data assets for any user question — using a combination of keyword matching, vector similarity, and graph traversal.
## Technical Details
* **Deployment**: EC2 instance in the shared platform account.
* **Access**: Cypher queries via GraphQL OGM (Object-Graph Mapping) at `api.{env}.brighthive.net/ogm`.
* **Security**: VPC isolation with encrypted connections. Access controlled via platform API authentication.
Comprehensive documentation for Neo4j database and Cypher query language
Implementation guide for GraphRAG using Neo4j knowledge graphs
# File Upload
Source: https://docs.brighthive.io/platform/file_upload
Upload data files to S3 with automatic schema detection and catalog registration in Neo4j.
## Overview
Brighthive makes it easy to get your data into the platform. Upload files to your organization's dedicated S3 data lake, and the platform automatically detects the schema, catalogs the data in Neo4j, and makes it available for querying.
## How It Works
1. **Upload your file** — Through the webapp or directly to your organization's S3 bucket.
2. **S3 stores the file** — Each organization gets dedicated S3 buckets (`brighthive-raw/`, `brighthive-staged/`, `brighthive-shared/`) in their own AWS account.
3. **Glue crawlers detect the schema** — Automatically infer column names, data types, partitions, and file format.
4. **Metadata registered in Neo4j** — Schema, row counts, location, and relationships are stored in the knowledge graph.
5. **Ready for querying** — Data is immediately available via Redshift Spectrum, BrightAgent, and the webapp data catalog.
## Supported Formats
* **Tabular**: CSV, Parquet, JSON, Avro, ORC, Excel
* **Documents**: PDF
* **Media**: Images, Videos
Tabular files are automatically schema-detected and made queryable. Documents and media are stored and cataloged for reference.
## Data Catalog
Every uploaded file is represented as a node in Neo4j's knowledge graph:
* **Schema** — Column names, data types, and partition structure.
* **Relationships** — Which organization owns it, which workspaces can access it.
* **Lineage** — How the data was uploaded and any transformations applied to it.
* **Metadata** — File size, row count, last updated timestamp, and custom tags.
The data catalog is accessible through both the webapp UI and BrightAgent — ask *"What data do we have about customers?"* and the Retrieval Agent searches Neo4j to find matching assets.
## Storage Architecture
Each organization gets isolated S3 storage in their dedicated AWS account:
* `brighthive-raw/` — Original uploaded files.
* `brighthive-staged/` — Processed and cleaned data.
* `brighthive-shared/` — Data shared with workspace services.
Cross-account access from workspace Redshift is handled securely via the `OrgDataCatalogRole` IAM role — no credentials are shared.
# Ingestion
Source: https://docs.brighthive.io/platform/ingestion
Upload files directly to S3 or connect 300+ sources via Airbyte — with automatic schema discovery and catalog registration.
## Overview
Brighthive supports two primary ingestion paths: direct S3 upload with automatic schema discovery, and Airbyte connectors for external data sources. Both paths automatically register metadata in Neo4j, making ingested data immediately discoverable by BrightAgent and the webapp.
## Direct S3 Upload
The primary ingestion path for most organizations. Upload files to your dedicated S3 data lake and the platform handles everything else:
```mermaid theme={null}
graph LR
A[Upload to S3] --> B[EventBridge Trigger]
B --> C[Step Functions]
C --> D[Glue Crawler]
D --> E[Schema Discovery]
E --> F[Neo4j Metadata Sync]
F --> G[Available for Querying]
```
1. **Upload** — Files land in the organization's S3 data lake (`brighthive-raw/` bucket).
2. **EventBridge detects the upload** — S3 `ObjectCreated` events trigger an EventBridge rule.
3. **Step Functions orchestrate ingestion** — The `DataIngestionStateMachine` coordinates the pipeline.
4. **Glue crawlers discover the schema** — Automatically infers column names, data types, partitions, and format.
5. **Metadata syncs to Neo4j** — A Lambda function updates the platform's knowledge graph with schema, row counts, and location.
6. **Data is queryable** — Immediately available via Redshift Spectrum external tables and BrightAgent.
### Supported File Formats
* CSV, Parquet, JSON, Avro, ORC, Excel
## Airbyte (Optional)
For organizations that need to ingest data from external systems, Brighthive offers a self-hosted Airbyte instance:
Connect to Shopify, HubSpot, PostgreSQL, MySQL, Salesforce, Google Analytics, and hundreds more sources.
Sources and connections are created programmatically through the GraphQL API — no manual Airbyte configuration needed.
Configure sync schedules per source — hourly, daily, or on-demand.
Runs on EC2 within the organization's dedicated AWS account for data isolation and security.
## Glue Data Catalog
AWS Glue is central to the ingestion pipeline:
* **Crawlers** automatically run when new data arrives, detecting schema changes and new partitions.
* **Data Catalog** stores table schemas, column metadata, and S3 locations.
* **Cross-account access** — Workspace Redshift reads the organization's Glue catalog via the `OrgDataCatalogRole` IAM role.
## What Happens After Ingestion
Once data is ingested and cataloged:
* **Neo4j** has full metadata — schema, location, row count, last updated timestamp.
* **Redshift** can query the data via Spectrum external tables.
* **BrightAgent** can discover and analyze the data through natural language.
* **Snowflake sync** is triggered if the organization uses Snowflake (via the `SnowflakeIngestionStateMachine`).
# Security
Source: https://docs.brighthive.io/platform/security
How Brighthive isolates your data, secures access, and protects your infrastructure at every layer.
## Overview
Security is foundational to Brighthive's architecture. Every customer gets dedicated, isolated AWS accounts — not shared tenancy. Your data never co-mingles with other customers, and every access path is authenticated, authorized, and auditable.
## Account Isolation
Brighthive uses AWS account boundaries as the primary isolation mechanism:
```mermaid theme={null}
graph TD
subgraph platform["Platform Account - Shared"]
API[GraphQL API]
Neo4j[Neo4j Metadata]
Cognito[Cognito Auth]
end
subgraph workspace["Workspace Account - Dedicated per Customer"]
Redshift[Redshift Serverless]
DBT[DBT Cloud]
end
subgraph org["Org Account - Dedicated per Organization"]
S3[S3 Data Lake]
Glue[Glue Catalog]
end
API --> Neo4j
API --> Cognito
Redshift -->|Cross-account IAM| S3
Redshift -->|Cross-account IAM| Glue
```
* **Organization accounts** are dedicated AWS accounts, each with their own VPC, S3 buckets, and Glue catalog. Your raw data lives here.
* **Workspace accounts** are dedicated AWS accounts, each with their own VPC and Redshift Serverless cluster. Your analytics run here.
* **Platform account** runs shared services (API, Neo4j, Cognito) that coordinate across customer accounts.
No customer data is stored in the shared platform account. Neo4j stores only metadata (schema, lineage, relationships) — never the actual data.
## Network Isolation
Every organization and workspace gets its own VPC with private subnets for data processing. No shared networking.
Data processing and storage run in private subnets. No public internet access to your data infrastructure.
Cross-account access uses VPC endpoints and IAM roles — not public internet or VPN tunnels.
Redshift Serverless runs across 3 availability zones for high availability within your dedicated VPC.
## Authentication & Authorization
### User Authentication
* **AWS Cognito** manages all user authentication with two separate pools: Platform (customer users) and Internal (admin).
* **JWT tokens** authenticate every API request. Tokens are validated by API Gateway custom authorizers before reaching any backend service.
* **Session management** ensures tokens expire and require re-authentication.
### Cross-Account Authorization
* **OrgDataCatalogRole** — An IAM role in each organization account that trusts the workspace's Redshift role. This is how Redshift queries organization data without sharing credentials.
* **No credentials are shared** — All cross-account access uses IAM role assumption (AWS STS), not stored passwords or API keys.
* **Workspace-scoped access** — The GraphQL API uses Neo4j metadata to route queries only to workspaces and organizations the user is authorized for.
## Data Isolation
| Layer | Isolation Mechanism |
| --------------- | ------------------------------------------------- |
| **AWS Account** | Dedicated accounts per organization and workspace |
| **Network** | Dedicated VPCs with private subnets |
| **Storage** | Dedicated S3 buckets per organization |
| **Warehouse** | Schema-per-organization in Redshift |
| **Metadata** | Neo4j access control lists |
| **Auth** | Cognito user pools with workspace membership |
## How Data Moves
Data flows through the platform via secure, authenticated paths:
### Ingestion (data in)
```
Your File → Org S3 (your AWS account) → Glue Crawler → Schema in Glue Catalog → Metadata synced to Neo4j
```
Your data lands in your dedicated S3 bucket in your own AWS account. It never leaves your account boundary during ingestion.
### Querying (data out)
```
User → Cognito Auth → GraphQL API → Neo4j (metadata lookup) → Redshift (your workspace) → Assumes OrgDataCatalogRole → Reads Org S3 via Spectrum
```
Redshift in your workspace assumes a cross-account IAM role to read data directly from the organization's S3. No data is copied — Spectrum reads in place.
### AI Agent Access
```
User → BrightAgent → Neo4j (discover data) → Redshift (query via workspace role) → Results
```
BrightAgent accesses data through the same authenticated paths as the webapp. Agents can only access data the user's workspace is authorized for.
## Secrets Management
* **AWS Secrets Manager** stores all credentials (database passwords, API keys, service tokens).
* **Snowflake JWT tokens** for Snowflake authentication — no stored passwords.
* **Environment variables** for non-sensitive configuration, managed per deployment environment.
* No secrets are stored in code, configuration files, or Neo4j.
## Deployment Security
* **Infrastructure as Code** — All infrastructure is managed via AWS CDK. No manual configuration or console changes.
* **CI/CD** — GitHub Actions and AWS CodeBuild for automated deployments with approval gates.
* **Three environments** — Dev, staging, and production with promotion workflows.
* **Primary region**: `us-east-1` with multi-AZ redundancy for critical services.
## Monitoring & Compliance
* **CloudWatch** — Logs and metrics for all Lambda functions and API performance.
* **Sentry** — Error tracking across platform services.
* **Neo4j audit logs** — All metadata access and modifications are logged.
* **Redshift query logs** — Full audit trail of every query executed against your warehouse.
For compliance certifications and detailed security documentation, visit our [Trust Center](https://trust.brighthive.io).
# Transformation
Source: https://docs.brighthive.io/platform/transformation
DBT Cloud via Datapiary for data transformations, with AI-generated models submitted as GitHub PRs.
## Overview
Brighthive uses DBT Cloud (integrated via Datapiary) for data transformations. The platform's DBT Agent can generate transformation models from natural language descriptions and submit them as GitHub pull requests — giving your team full control over what gets deployed.
## How It Works
```mermaid theme={null}
graph TD
A[Describe Transformation] --> B[DBT Agent Generates Model]
B --> C[GitHub PR Created]
C --> D[Team Reviews & Approves]
D --> E[DBT Cloud Runs Model]
E --> F[Results in Redshift]
F --> G[Lineage Tracked in Neo4j]
```
1. **Describe what you need** — Tell BrightAgent what transformation you want in plain English.
2. **DBT Agent generates the model** — Creates SQL transformations with proper dbt structure, configurations, and tests.
3. **GitHub PR submitted** — All generated code goes through a pull request for human review.
4. **Team reviews and approves** — Nothing gets deployed without your approval.
5. **DBT Cloud executes** — Approved models run on DBT Cloud, executing transformations in your Redshift warehouse.
6. **Neo4j tracks lineage** — Transformation relationships are recorded in the knowledge graph.
## DBT Cloud Integration
DBT Cloud is integrated through the Datapiary library, providing a consistent interface for transformation management.
dbt models execute in your workspace's dedicated Redshift Serverless cluster.
All dbt models live in Git — full history, branching, and code review via GitHub PRs.
Neo4j tracks dependencies between source tables, staging layers, and data marts — visible in the data catalog.
## What Neo4j Tracks
Neo4j serves as the control plane for transformations, maintaining visibility across:
* **Source tables** — Raw data from organizations.
* **Staging layers** — Cleaned and standardized intermediate tables.
* **Data marts** — Final analytical tables ready for consumption.
* **Dependencies** — Which models depend on which sources and other models.
* **Run history** — When transformations last ran and their status.
# Warehouse
Source: https://docs.brighthive.io/platform/warehouse
Redshift Serverless and Snowflake provide auto-scaling, isolated data warehousing for every workspace.
## Overview
Every Brighthive workspace gets a dedicated data warehouse deployed in its own AWS account. Redshift Serverless is the primary warehouse, with Snowflake available via Datapiary for organizations that need it.
## Redshift Serverless
Serverless compute scales automatically based on query workload — no capacity planning or cluster management required.
Deployed across 3 AZs for high availability and fault tolerance within each workspace's dedicated VPC.
Each organization's data lives in its own Redshift schema, providing logical isolation within the shared workspace warehouse.
Lambda-backed REST API enables the platform and BrightAgent to execute queries programmatically against your warehouse.
### Cross-Account Data Access
Redshift in your workspace account queries organization data stored in separate AWS accounts using cross-account IAM roles:
```
Workspace Redshift → Assumes OrgDataCatalogRole → Reads Org S3 + Glue Catalog
```
* **OrgDataCatalogRole** is an IAM role in each organization's account that trusts the workspace's Redshift role.
* Redshift Spectrum queries S3 data directly via external tables — no data copying required.
* Glue Data Catalog provides schema metadata for these external tables.
### Redshift Spectrum
Redshift Spectrum enables querying data directly in S3 without loading it into Redshift tables. This is used for:
* Querying large datasets that don't need to be materialized in the warehouse.
* Accessing the latest organization data immediately after upload (via Glue catalog references).
## Snowflake (via Datapiary)
For organizations that need Snowflake alongside Redshift, Brighthive provides Snowflake integration through Datapiary:
* Organizations can sync data from their S3 data lake to Snowflake.
* Snowflake assumes the OrgDataCatalogRole to access organization S3 data via cross-account IAM.
* DBT Cloud transformations can run against Snowflake in addition to Redshift.
## How Data Gets Into Your Warehouse
1. **Organization uploads data** to their S3 data lake.
2. **Glue crawlers** auto-detect the schema and update the Glue Data Catalog.
3. **Redshift Spectrum** creates external tables pointing to the organization's S3 and Glue catalog.
4. **Metadata is synced** to Neo4j, making the data discoverable by BrightAgent and the webapp.
5. **Optionally**, data is synced to Snowflake for organizations that use it.
# WebApp
Source: https://docs.brighthive.io/platform/webapp
React-based webapp serving as the Brighthive platform UI with data catalog, BrightAgent, collaboration, and configuration.
## Overview
The Brighthive webapp is a React application that serves as the central interface for all platform features. Built with Apollo GraphQL Client for real-time data fetching and Stream.io for team collaboration.
## Core Sections
Browse and search your data assets. See schemas, metadata, lineage, and quality scores — all powered by the Neo4j knowledge graph.
Conversational AI interface for querying data, generating visualizations, building dbt models, and exploring your data estate through natural language.
Onboard data through file uploads to S3 or by configuring Airbyte connectors for external data sources.
Define schemas, manage data policies, explore the data catalog, and review data quality metrics.
Create projects, manage workflows, and access BrightAgent Studio for custom agent configurations.
Platform settings including ingestion sources (Airbyte), warehouse connections (Redshift/Snowflake), and transformation configs (DBT Cloud).
## Technology Stack
* **React + TypeScript** — Component-based UI with static typing.
* **Apollo Client** — GraphQL data fetching and state management, connected to the platform's Apollo Federation API.
* **Material-UI + TailwindCSS** — UI component library and utility-first styling.
* **Stream.io** — Real-time team collaboration and chat embedded in the platform.
* **AG Grid Enterprise** — High-performance data tables for browsing large datasets.
* **Monaco Editor** — Code editing interface for reviewing generated SQL, dbt models, and notebooks.
## Real-time Collaboration
Stream.io integration enables team collaboration directly within the platform:
* Chat with team members while exploring data.
* Share BrightAgent insights and visualizations.
* Discuss data quality issues and governance decisions in context.
## Deployment
* Hosted on **AWS Amplify** with dev, staging, and production environments.
* Authenticates via **Cognito** (JWT tokens) against the platform's user pools.
* Communicates with the backend via **GraphQL** (`REACT_APP_GRAPHQL_URL`) and BrightAgent via its API (`REACT_APP_BB_ASSISTANT_URL`).