auth-yes/forum/DATA_STRUCTURES.md

Agent Forum v4 - Fundamental Data Structures Reference

This document serves as the comprehensive list and reference for all data structures, data sources, and embedded storage mechanisms outlined in the agent-forum-v4 blueprint. The goal is to provide a unified overview of the machine-readable structures that agents will interact with, entirely eliminating the need for external cloud SaaS databases.

1. Storage Layers (Git-Native Storage)

1.0 Protocol Buffers (Protobuf)

• Purpose: Facilitates high-performance, conversion-less data transfer between agents.
• Content: Serialized binary representations of agent state, telemetry, and index data.
• Integration: Works natively with SCIP indexes and TurboQuant compressed vector math to drastically reduce I/O latency.

1.1 Git Notes (refs/notes/commits)

• Purpose: Attaches arbitrary metadata directly to Git commits without altering the commit hash or polluting the working directory.
• Content: Primarily JSON payloads containing:
  • Agent "meta-thoughts" and reasoning.
  • Risk assessments (e.g., generated by the Adversary).
  • Telemetry summaries related to a specific commit.
• Example Fetch: git log --show-notes="forum/reasoning"

1.2 Orphan Branches (Meta-State Branch)

• Purpose: An isolated Git branch (e.g., forum/meta-state) that tracks ongoing project state separately from the main source code. It shares no commit history with the main branch.
• Content:
  • CI/CD telemetry JSONs.
  • Requirements Traceability Matrices (RTM).
  • The Project Task DAG files.
  • Periodic serialized graphs (e.g., SQLite dumps or graph snapshots).

1.3 Embedded Vector Databases (sqlite-vec)

• Purpose: Provides fuzzy, associative memory retrieval without a dedicated network vector database. Compresses large documents via Locality-Sensitive Hashing (LSH) and HNSW.
• Content: Highly compressed, quantized embeddings of concepts (PRDs, ADRs, documentation).
• Structure: Isolated SQLite files (e.g., docs_graph.sqlite, telemetry_graph.sqlite) to prevent cross-contamination of semantic data.

1.4 TurboQuant

• Purpose: Compresses high-dimensional semantic concepts into binary hashes using 2-bit to 4-bit quantization.
• Content: Extremely lightweight local embedded indexes (often under 30MB) facilitating millisecond vector search inside sqlite-vec.

2. Process & Governance Structures

2.0 Declarative Frontmatter (YAML UUIDs)

• Purpose: Uniquely identifies Markdown artifacts to maintain traceability within the project DAG and the vector databases.
• Content: YAML blocks containing a unique UUID (Artifact-ID).
• Format Note: MUST be compatible with UUIDv7 (time-ordered) to allow historical sorting and chronological sequence inference directly from the identifier, acting as a strict primary key.

2.1 The Project DAG (YAML)

• Purpose: Replaces traditional flat project management tools (like Jira or Markdown task lists). Dictates execution order mathematically.
• Content: YAML files representing a Directed Acyclic Graph.
• Key Fields:
  • id: A UUIDv7 acting as the unique identifier.
  • blocked_by: Array of UUIDs this task depends on.
  • legacy_slug (Optional): The visual human-readable string (e.g., YYYY-MMDD.[sequence]...).
  • status: e.g., pending, in_progress, completed.
  • description: The actual prompt/goal.

2.2 Bounded Model Checking (Transitions Matrix)

• Purpose: Defines strict state machine rules for the agent pipeline to guarantee proper governance.
• Format: transitions.json
• Content: A JSON mapping that states which roles can execute under which conditions (e.g., "Coder": { "requires": ["Gatekeeper_Approval"] }).

2.3 The Constitution (AGENTS.md)

• Purpose: The supreme machine-readable ruleset that all agents must ingest to understand the target application stack, constraints, and operational boundaries.

3. Code Intelligence Structures

3.0 Git Merkle DAG Diffing

• Purpose: Ensures O(1) context updates for agents by identifying exact modified file hashes without reading raw file strings.
• Content: Hashes resulting from zero-overhead diffing (e.g., git ls-tree and git diff-tree).

3.1 SCIP Indexes (Semantic Code Intelligence Protocol)

• Purpose: Replaces unreliable regex-based searching with a statically guaranteed mapping of code symbols.
• Content: A lightweight database mapping definitions, references, and relationships across the codebase. Extracted typically via Tree-sitter.

3.2 Abstract Syntax Trees (ASTs) & Control Flow Graphs (CFGs)

• Purpose: Structured representations of code syntax and execution paths.
• Content: JSON/XML mapping of every possible path a variable can take, used by the Adversary agent to deterministically prove security flaws (e.g., unsanitized inputs reaching SQL statements).

3.3 Mutation Testing Scores

• Purpose: Represents the "blast radius" and effectiveness of test suites.
• Content: Structured outputs from tools like Stryker or Mutmut that indicate how many injected bugs were successfully caught by the test physics.

3.4 Dependency Graphing (Adjacency Matrices)

• Purpose: Mathematically calculates the exact "blast radius" of any code change.
• Content: Adjacency matrices (generated by tools like CodeSee or Madge) that map the downstream and upstream impact across components.

4. Semantic & Telemetry Structures

4.1 Ontologies (JSON-LD)

• Purpose: Replaces legacy requirements management (like DOORS). Achieves deep traceability by linking code/tasks to business requirements.
• Content: Linked Data JSON blocks embedded in documentation (@type: "Requirement"). These compile into a single mathematical ontology.graph file.

4.2 OpenTelemetry Traces (.trace.json)

• Purpose: Captures millisecond-level execution latencies during testing.
• Content: Standardized OTel trace JSON payloads ingested by the Adversary to find physical execution bottlenecks in the code.

4.3 Team Friction Telemetry

• Purpose: Used by the Analyst agent to measure the efficiency of human-to-agent collaboration.
• Content: JSON payloads stored in the meta-state branch recording metrics like Mean Time to Resolution (MTTR), PR comment-to-code ratios, and idle handoff durations.