Tyler Gillispie
4df94c2a19
- Updates `forum/DATA_STRUCTURES.md` with missing concepts: Protocol Buffers, TurboQuant, Git Merkle DAG Diffing, Dependency Graphing, and Declarative Frontmatter (UUIDv7). - Expands `forum/experiments/lab.ts` with 5 new proofs-of-concept for the new data structures. - Adds `protobuf_poc.ts`, `merkle_diff_poc.ts`, `vector_db_poc.ts`, `dependency_graph_poc.ts`, and `telemetry_poc.ts`. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: mrteye <1945243+mrteye@users.noreply.github.com>
101 lines
6.3 KiB
Markdown
101 lines
6.3 KiB
Markdown
# Agent Forum v4 - Fundamental Data Structures Reference
|
|
|
|
This document serves as the comprehensive list and reference for all data structures, data sources, and embedded storage mechanisms outlined in the `agent-forum-v4` blueprint. The goal is to provide a unified overview of the machine-readable structures that agents will interact with, entirely eliminating the need for external cloud SaaS databases.
|
|
|
|
## 1. Storage Layers (Git-Native Storage)
|
|
|
|
### 1.0 Protocol Buffers (Protobuf)
|
|
- **Purpose**: Facilitates high-performance, conversion-less data transfer between agents.
|
|
- **Content**: Serialized binary representations of agent state, telemetry, and index data.
|
|
- **Integration**: Works natively with SCIP indexes and TurboQuant compressed vector math to drastically reduce I/O latency.
|
|
|
|
|
|
### 1.1 Git Notes (`refs/notes/commits`)
|
|
- **Purpose**: Attaches arbitrary metadata directly to Git commits without altering the commit hash or polluting the working directory.
|
|
- **Content**: Primarily JSON payloads containing:
|
|
- Agent "meta-thoughts" and reasoning.
|
|
- Risk assessments (e.g., generated by the Adversary).
|
|
- Telemetry summaries related to a specific commit.
|
|
- **Example Fetch**: `git log --show-notes="forum/reasoning"`
|
|
|
|
### 1.2 Orphan Branches (Meta-State Branch)
|
|
- **Purpose**: An isolated Git branch (e.g., `forum/meta-state`) that tracks ongoing project state separately from the main source code. It shares no commit history with the main branch.
|
|
- **Content**:
|
|
- CI/CD telemetry JSONs.
|
|
- Requirements Traceability Matrices (RTM).
|
|
- The Project Task DAG files.
|
|
- Periodic serialized graphs (e.g., SQLite dumps or graph snapshots).
|
|
|
|
### 1.3 Embedded Vector Databases (`sqlite-vec`)
|
|
- **Purpose**: Provides fuzzy, associative memory retrieval without a dedicated network vector database. Compresses large documents via Locality-Sensitive Hashing (LSH) and HNSW.
|
|
- **Content**: Highly compressed, quantized embeddings of concepts (PRDs, ADRs, documentation).
|
|
- **Structure**: Isolated SQLite files (e.g., `docs_graph.sqlite`, `telemetry_graph.sqlite`) to prevent cross-contamination of semantic data.
|
|
|
|
### 1.4 TurboQuant
|
|
- **Purpose**: Compresses high-dimensional semantic concepts into binary hashes using 2-bit to 4-bit quantization.
|
|
- **Content**: Extremely lightweight local embedded indexes (often under 30MB) facilitating millisecond vector search inside `sqlite-vec`.
|
|
|
|
|
|
## 2. Process & Governance Structures
|
|
|
|
### 2.0 Declarative Frontmatter (YAML UUIDs)
|
|
- **Purpose**: Uniquely identifies Markdown artifacts to maintain traceability within the project DAG and the vector databases.
|
|
- **Content**: YAML blocks containing a unique UUID (Artifact-ID).
|
|
- **Format Note**: MUST be compatible with UUIDv7 (time-ordered) to allow historical sorting and chronological sequence inference directly from the identifier, acting as a strict primary key.
|
|
|
|
|
|
### 2.1 The Project DAG (YAML)
|
|
- **Purpose**: Replaces traditional flat project management tools (like Jira or Markdown task lists). Dictates execution order mathematically.
|
|
- **Content**: YAML files representing a Directed Acyclic Graph.
|
|
- **Key Fields**:
|
|
- `id`: A UUIDv7 acting as the unique identifier.
|
|
- `blocked_by`: Array of UUIDs this task depends on.
|
|
- `legacy_slug` (Optional): The visual human-readable string (e.g., `YYYY-MMDD.[sequence]...`).
|
|
- `status`: e.g., `pending`, `in_progress`, `completed`.
|
|
- `description`: The actual prompt/goal.
|
|
|
|
### 2.2 Bounded Model Checking (Transitions Matrix)
|
|
- **Purpose**: Defines strict state machine rules for the agent pipeline to guarantee proper governance.
|
|
- **Format**: `transitions.json`
|
|
- **Content**: A JSON mapping that states which roles can execute under which conditions (e.g., `"Coder": { "requires": ["Gatekeeper_Approval"] }`).
|
|
|
|
### 2.3 The Constitution (`AGENTS.md`)
|
|
- **Purpose**: The supreme machine-readable ruleset that all agents must ingest to understand the target application stack, constraints, and operational boundaries.
|
|
|
|
## 3. Code Intelligence Structures
|
|
|
|
### 3.0 Git Merkle DAG Diffing
|
|
- **Purpose**: Ensures O(1) context updates for agents by identifying exact modified file hashes without reading raw file strings.
|
|
- **Content**: Hashes resulting from zero-overhead diffing (e.g., `git ls-tree` and `git diff-tree`).
|
|
|
|
|
|
### 3.1 SCIP Indexes (Semantic Code Intelligence Protocol)
|
|
- **Purpose**: Replaces unreliable regex-based searching with a statically guaranteed mapping of code symbols.
|
|
- **Content**: A lightweight database mapping definitions, references, and relationships across the codebase. Extracted typically via Tree-sitter.
|
|
|
|
### 3.2 Abstract Syntax Trees (ASTs) & Control Flow Graphs (CFGs)
|
|
- **Purpose**: Structured representations of code syntax and execution paths.
|
|
- **Content**: JSON/XML mapping of every possible path a variable can take, used by the Adversary agent to deterministically prove security flaws (e.g., unsanitized inputs reaching SQL statements).
|
|
|
|
### 3.3 Mutation Testing Scores
|
|
- **Purpose**: Represents the "blast radius" and effectiveness of test suites.
|
|
- **Content**: Structured outputs from tools like Stryker or Mutmut that indicate how many injected bugs were successfully caught by the test physics.
|
|
|
|
### 3.4 Dependency Graphing (Adjacency Matrices)
|
|
- **Purpose**: Mathematically calculates the exact "blast radius" of any code change.
|
|
- **Content**: Adjacency matrices (generated by tools like CodeSee or Madge) that map the downstream and upstream impact across components.
|
|
|
|
## 4. Semantic & Telemetry Structures
|
|
|
|
### 4.1 Ontologies (JSON-LD)
|
|
- **Purpose**: Replaces legacy requirements management (like DOORS). Achieves deep traceability by linking code/tasks to business requirements.
|
|
- **Content**: Linked Data JSON blocks embedded in documentation (`@type: "Requirement"`). These compile into a single mathematical `ontology.graph` file.
|
|
|
|
### 4.2 OpenTelemetry Traces (`.trace.json`)
|
|
- **Purpose**: Captures millisecond-level execution latencies during testing.
|
|
- **Content**: Standardized OTel trace JSON payloads ingested by the Adversary to find physical execution bottlenecks in the code.
|
|
|
|
### 4.3 Team Friction Telemetry
|
|
- **Purpose**: Used by the Analyst agent to measure the efficiency of human-to-agent collaboration.
|
|
- **Content**: JSON payloads stored in the meta-state branch recording metrics like Mean Time to Resolution (MTTR), PR comment-to-code ratios, and idle handoff durations.
|