# Agent Forum v4 - Fundamental Data Structures Reference

This document serves as the comprehensive list and reference for all data
structures, data sources, and embedded storage mechanisms outlined in the
`agent-forum-v4` blueprint. The goal is to provide a unified overview of the
machine-readable structures that agents will interact with, entirely eliminating
the need for external cloud SaaS databases.

## 1. Storage Layers (Git-Native Storage)

### 1.0 Protocol Buffers (Protobuf)

- **Purpose**: Facilitates high-performance, conversion-less data transfer
  between agents.
- **Content**: Serialized binary representations of agent state, telemetry, and
  index data.
- **Integration**: Works natively with SCIP indexes and TurboQuant compressed
  vector math to drastically reduce I/O latency.

### 1.1 Git Notes (`refs/notes/commits`)

- **Purpose**: Attaches arbitrary metadata directly to Git commits without
  altering the commit hash or polluting the working directory.
- **Content**: Primarily JSON payloads containing:
  - Agent "meta-thoughts" and reasoning.
  - Risk assessments (e.g., generated by the Adversary).
  - Telemetry summaries related to a specific commit.
- **Example Fetch**: `git log --show-notes="forum/reasoning"`

### 1.2 Orphan Branches (Meta-State Branch)

- **Purpose**: An isolated Git branch (e.g., `forum/meta-state`) that tracks
  ongoing project state separately from the main source code. It shares no
  commit history with the main branch.
- **Content**:
  - CI/CD telemetry JSONs.
  - Requirements Traceability Matrices (RTM).
  - The Project Task DAG files.
  - Periodic serialized graphs (e.g., SQLite dumps or graph snapshots).

### 1.3 Embedded Vector Databases (`sqlite-vec`)

- **Purpose**: Provides fuzzy, associative memory retrieval without a dedicated
  network vector database. Compresses large documents via Locality-Sensitive
  Hashing (LSH) and HNSW.
- **Content**: Highly compressed, quantized embeddings of concepts (PRDs, ADRs,
  documentation).
- **Structure**: Isolated SQLite files (e.g., `docs_graph.sqlite`,
  `telemetry_graph.sqlite`) to prevent cross-contamination of semantic data.

### 1.4 TurboQuant

- **Purpose**: Compresses high-dimensional semantic concepts into binary hashes
  using 2-bit to 4-bit quantization.
- **Content**: Extremely lightweight local embedded indexes (often under 30MB)
  facilitating millisecond vector search inside `sqlite-vec`.

## 2. Process & Governance Structures

### 2.0 Declarative Frontmatter (YAML UUIDs)

- **Purpose**: Uniquely identifies Markdown artifacts to maintain traceability
  within the project DAG and the vector databases.
- **Content**: YAML blocks containing a unique UUID (Artifact-ID).
- **Format Note**: MUST be compatible with UUIDv7 (time-ordered) to allow
  historical sorting and chronological sequence inference directly from the
  identifier, acting as a strict primary key.

### 2.1 The Project DAG (YAML)

- **Purpose**: Replaces traditional flat project management tools (like Jira or
  Markdown task lists). Dictates execution order mathematically.
- **Content**: YAML files representing a Directed Acyclic Graph.
- **Key Fields**:
  - `id`: A UUIDv7 acting as the unique identifier.
  - `blocked_by`: Array of UUIDs this task depends on.
  - `legacy_slug` (Optional): The visual human-readable string (e.g.,
    `YYYY-MMDD.[sequence]...`).
  - `status`: e.g., `pending`, `in_progress`, `completed`.
  - `description`: The actual prompt/goal.

### 2.2 Bounded Model Checking (Transitions Matrix)

- **Purpose**: Defines strict state machine rules for the agent pipeline to
  guarantee proper governance.
- **Format**: `transitions.json`
- **Content**: A JSON mapping that states which roles can execute under which
  conditions (e.g., `"Coder": { "requires": ["Gatekeeper_Approval"] }`).

### 2.3 The Constitution (`AGENTS.md`)

- **Purpose**: The supreme machine-readable ruleset that all agents must ingest
  to understand the target application stack, constraints, and operational
  boundaries.

## 3. Code Intelligence Structures

### 3.0 Git Merkle DAG Diffing

- **Purpose**: Ensures O(1) context updates for agents by identifying exact
  modified file hashes without reading raw file strings.
- **Content**: Hashes resulting from zero-overhead diffing (e.g., `git ls-tree`
  and `git diff-tree`).

### 3.1 SCIP Indexes (Semantic Code Intelligence Protocol)

- **Purpose**: Replaces unreliable regex-based searching with a statically
  guaranteed mapping of code symbols.
- **Content**: A lightweight database mapping definitions, references, and
  relationships across the codebase. Extracted typically via Tree-sitter.

### 3.2 Abstract Syntax Trees (ASTs) & Control Flow Graphs (CFGs)

- **Purpose**: Structured representations of code syntax and execution paths.
- **Content**: JSON/XML mapping of every possible path a variable can take, used
  by the Adversary agent to deterministically prove security flaws (e.g.,
  unsanitized inputs reaching SQL statements).

### 3.3 Mutation Testing Scores

- **Purpose**: Represents the "blast radius" and effectiveness of test suites.
- **Content**: Structured outputs from tools like Stryker or Mutmut that
  indicate how many injected bugs were successfully caught by the test physics.

### 3.4 Dependency Graphing (Adjacency Matrices)

- **Purpose**: Mathematically calculates the exact "blast radius" of any code
  change.
- **Content**: Adjacency matrices (generated by tools like CodeSee or Madge)
  that map the downstream and upstream impact across components.

## 4. Semantic & Telemetry Structures

### 4.1 Ontologies (JSON-LD)

- **Purpose**: Replaces legacy requirements management (like DOORS). Achieves
  deep traceability by linking code/tasks to business requirements.
- **Content**: Linked Data JSON blocks embedded in documentation
  (`@type: "Requirement"`). These compile into a single mathematical
  `ontology.graph` file.

### 4.2 OpenTelemetry Traces (`.trace.json`)

- **Purpose**: Captures millisecond-level execution latencies during testing.
- **Content**: Standardized OTel trace JSON payloads ingested by the Adversary
  to find physical execution bottlenecks in the code.

### 4.3 Team Friction Telemetry

- **Purpose**: Used by the Analyst agent to measure the efficiency of
  human-to-agent collaboration.
- **Content**: JSON payloads stored in the meta-state branch recording metrics
  like Mean Time to Resolution (MTTR), PR comment-to-code ratios, and idle
  handoff durations.