data-agents-copilot: a multi-agent system for data

Problem

Data engineering tasks (SQL, PySpark, pipelines, governance) require very different contexts and standards. A single generic assistant gets naming wrong, ignores best practices and loses context between interactions.

Architecture

flowchart TB
  U[User - CLI / VS Code / Web] --> R[Router / Supervisor]
  R --> A1[Spark Expert]
  R --> A2[SQL Expert]
  R --> A3[Pipeline Architect]
  R --> A4[Data Quality]
  R --> A5[Governance]
  A1 --> KB[(Knowledge Base + Memory)]
  A2 --> KB
  A3 --> KB
  A4 --> KB
  A5 --> KB
  R --> MCP[MCP Servers]
  classDef hub fill:#0d1525,stroke:#3b82f6,color:#e2e8f0
  classDef store fill:#0d1525,stroke:#10b981,color:#e2e8f0
  class U,R,A1,A2,A3,A4,A5,MCP hub
  class KB store

Solution

• Router that identifies the task domain and dispatches to the right specialist agent.
• 15 specialized agents (Spark, SQL, Pipeline, Data Quality, Governance, etc.).
• Automatic naming governance and collaborative multi-agent workflows.
• Structured Knowledge Base, episodic memory and peer-to-peer QA protocol.
• Runs via CLI, Chainlit (web) or directly in VS Code, with MCP servers.

Results

• Specialized answers consistent with each domain's standards.
• Context preserved across interactions via memory.
• Extensible foundation for new agents and integrations.

Stack