Tyler Gillispie
f3fe8b77e6
- Updates CONCEPTS.md and DATA_STRUCTURES.md to include Multi-Vec Isolation, Orchestration Matrix, and Static Analysis Payloads. - Adds GRAVEYARD.md to document dismissed anti-patterns (Doc-to-LoRA and PASTE). - Implements corresponding Proof-of-Concept scripts in `forum/experiments/` (multi_vec_poc.ts, orchestration_matrix_poc.ts, static_analysis_poc.ts, and graveyard_poc.ts). - Integrates all new PoCs into the `lab.ts` experiment runner. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: mrteye <1945243+mrteye@users.noreply.github.com>
176 lines
7.5 KiB
Markdown
176 lines
7.5 KiB
Markdown
# Agent Forum v4 - Fundamental Data Structures Reference
|
|
|
|
This document serves as the comprehensive list and reference for all data
|
|
structures, data sources, and embedded storage mechanisms outlined in the
|
|
`agent-forum-v4` blueprint. The goal is to provide a unified overview of the
|
|
machine-readable structures that agents will interact with, entirely eliminating
|
|
the need for external cloud SaaS databases.
|
|
|
|
## 1. Storage Layers (Git-Native Storage)
|
|
|
|
### 1.0 Protocol Buffers (Protobuf)
|
|
|
|
- **Purpose**: Facilitates high-performance, conversion-less data transfer
|
|
between agents.
|
|
- **Content**: Serialized binary representations of agent state, telemetry, and
|
|
index data.
|
|
- **Integration**: Works natively with SCIP indexes and TurboQuant compressed
|
|
vector math to drastically reduce I/O latency.
|
|
|
|
### 1.1 Git Notes (`refs/notes/commits`)
|
|
|
|
- **Purpose**: Attaches arbitrary metadata directly to Git commits without
|
|
altering the commit hash or polluting the working directory.
|
|
- **Content**: Primarily JSON payloads containing:
|
|
- Agent "meta-thoughts" and reasoning.
|
|
- Risk assessments (e.g., generated by the Adversary).
|
|
- Telemetry summaries related to a specific commit.
|
|
- **Example Fetch**: `git log --show-notes="forum/reasoning"`
|
|
|
|
### 1.2 Orphan Branches (Meta-State Branch)
|
|
|
|
- **Purpose**: An isolated Git branch (e.g., `forum/meta-state`) that tracks
|
|
ongoing project state separately from the main source code. It shares no
|
|
commit history with the main branch.
|
|
- **Content**:
|
|
- CI/CD telemetry JSONs.
|
|
- Requirements Traceability Matrices (RTM).
|
|
- The Project Task DAG files.
|
|
- Periodic serialized graphs (e.g., SQLite dumps or graph snapshots).
|
|
|
|
### 1.3 Embedded Vector Databases (`sqlite-vec`)
|
|
|
|
- **Purpose**: Provides fuzzy, associative memory retrieval without a dedicated
|
|
network vector database. Compresses large documents via Locality-Sensitive
|
|
Hashing (LSH) and HNSW.
|
|
- **Content**: Highly compressed, quantized embeddings of concepts (PRDs, ADRs,
|
|
documentation).
|
|
- **Structure**: Isolated SQLite files (e.g., `docs_graph.sqlite`,
|
|
`telemetry_graph.sqlite`) to prevent cross-contamination of semantic data.
|
|
|
|
### 1.4 TurboQuant
|
|
|
|
- **Purpose**: Compresses high-dimensional semantic concepts into binary hashes
|
|
using 2-bit to 4-bit quantization.
|
|
- **Content**: Extremely lightweight local embedded indexes (often under 30MB)
|
|
facilitating millisecond vector search inside `sqlite-vec`.
|
|
|
|
### 1.5 Multi-Vec Isolation
|
|
|
|
- **Purpose**: Prevents semantic bleed by utilizing isolated, separate `sqlite-vec` files rather than dumping all embeddings into a single vector database.
|
|
- **Content**: Domain-specific embedding files (e.g., `docs_graph.sqlite` distinct from `telemetry_graph.sqlite`).
|
|
|
|
## 2. Process & Governance Structures
|
|
|
|
### 2.0 Declarative Frontmatter (YAML UUIDs)
|
|
|
|
- **Purpose**: Uniquely identifies Markdown artifacts to maintain traceability
|
|
within the project DAG and the vector databases.
|
|
- **Content**: YAML blocks containing a unique UUID (Artifact-ID).
|
|
- **Format Note**: MUST be compatible with UUIDv7 (time-ordered) to allow
|
|
historical sorting and chronological sequence inference directly from the
|
|
identifier, acting as a strict primary key.
|
|
|
|
### 2.1 The Project DAG (YAML)
|
|
|
|
- **Purpose**: Replaces traditional flat project management tools (like Jira or
|
|
Markdown task lists). Dictates execution order mathematically.
|
|
- **Content**: YAML files representing a Directed Acyclic Graph.
|
|
- **Key Fields**:
|
|
- `id`: A UUIDv7 acting as the unique identifier.
|
|
- `blocked_by`: Array of UUIDs this task depends on.
|
|
- `legacy_slug` (Optional): The visual human-readable string (e.g.,
|
|
`YYYY-MMDD.[sequence]...`).
|
|
- `status`: e.g., `pending`, `in_progress`, `completed`.
|
|
- `description`: The actual prompt/goal.
|
|
|
|
### 2.2 Bounded Model Checking (Transitions Matrix)
|
|
|
|
- **Purpose**: Defines strict state machine rules for the agent pipeline to
|
|
guarantee proper governance.
|
|
- **Format**: `transitions.json`
|
|
- **Content**: A JSON mapping that states which roles can execute under which
|
|
conditions (e.g., `"Coder": { "requires": ["Gatekeeper_Approval"] }`).
|
|
|
|
### 2.3 The Constitution (`AGENTS.md`)
|
|
|
|
- **Purpose**: The supreme machine-readable ruleset that all agents must ingest
|
|
to understand the target application stack, constraints, and operational
|
|
boundaries.
|
|
|
|
### 2.4 The Orchestration Matrix (Agent Roles)
|
|
|
|
- **Purpose**: Defines the inputs, outputs, and primary directives of the 6 core agents.
|
|
- **Roles**:
|
|
- **Gatekeeper**: Translates Ontologies/DAGs to Verification Checklists.
|
|
- **Historian**: Uses sqlite-vec and Git Notes to inject historical context.
|
|
- **Adversary**: Reads SCIP, CFGs, and Mutation data to generate tests/mutations.
|
|
- **Translator**: Consumes SCIP diffs to output API references and docs.
|
|
- **Analyst**: Reads Telemetry to propose workflow optimizations.
|
|
- **Evaluator**: Governs pipeline progression by reading transitions.json.
|
|
|
|
## 3. Code Intelligence Structures
|
|
|
|
### 3.0 Git Merkle DAG Diffing
|
|
|
|
- **Purpose**: Ensures O(1) context updates for agents by identifying exact
|
|
modified file hashes without reading raw file strings.
|
|
- **Content**: Hashes resulting from zero-overhead diffing (e.g., `git ls-tree`
|
|
and `git diff-tree`).
|
|
|
|
### 3.1 SCIP Indexes (Semantic Code Intelligence Protocol)
|
|
|
|
- **Purpose**: Replaces unreliable regex-based searching with a statically
|
|
guaranteed mapping of code symbols.
|
|
- **Content**: A lightweight database mapping definitions, references, and
|
|
relationships across the codebase. Extracted typically via Tree-sitter.
|
|
|
|
### 3.2 Abstract Syntax Trees (ASTs) & Control Flow Graphs (CFGs)
|
|
|
|
- **Purpose**: Structured representations of code syntax and execution paths.
|
|
- **Content**: JSON/XML mapping of every possible path a variable can take, used
|
|
by the Adversary agent to deterministically prove security flaws (e.g.,
|
|
unsanitized inputs reaching SQL statements).
|
|
|
|
### 3.3 Mutation Testing Scores
|
|
|
|
- **Purpose**: Represents the "blast radius" and effectiveness of test suites.
|
|
- **Content**: Structured outputs from tools like Stryker or Mutmut that
|
|
indicate how many injected bugs were successfully caught by the test physics.
|
|
|
|
### 3.4 Dependency Graphing (Adjacency Matrices)
|
|
|
|
- **Purpose**: Mathematically calculates the exact "blast radius" of any code
|
|
change.
|
|
- **Content**: Adjacency matrices (generated by tools like CodeSee or Madge)
|
|
that map the downstream and upstream impact across components.
|
|
|
|
### 3.5 Static Analysis Payloads
|
|
|
|
- **Purpose**: Provides compiler-grade code smell and vulnerability metrics directly to triage agents.
|
|
- **Content**: Standardized JSON/XML outputs from industry tools (e.g., Semgrep, SonarQube).
|
|
|
|
## 4. Semantic & Telemetry Structures
|
|
|
|
### 4.1 Ontologies (JSON-LD)
|
|
|
|
- **Purpose**: Replaces legacy requirements management (like DOORS). Achieves
|
|
deep traceability by linking code/tasks to business requirements.
|
|
- **Content**: Linked Data JSON blocks embedded in documentation
|
|
(`@type: "Requirement"`). These compile into a single mathematical
|
|
`ontology.graph` file.
|
|
|
|
### 4.2 OpenTelemetry Traces (`.trace.json`)
|
|
|
|
- **Purpose**: Captures millisecond-level execution latencies during testing.
|
|
- **Content**: Standardized OTel trace JSON payloads ingested by the Adversary
|
|
to find physical execution bottlenecks in the code.
|
|
|
|
### 4.3 Team Friction Telemetry
|
|
|
|
- **Purpose**: Used by the Analyst agent to measure the efficiency of
|
|
human-to-agent collaboration.
|
|
- **Content**: JSON payloads stored in the meta-state branch recording metrics
|
|
like Mean Time to Resolution (MTTR), PR comment-to-code ratios, and idle
|
|
handoff durations.
|