diff --git a/forum/CONCEPTS.md b/forum/CONCEPTS.md index 1087de3..050cb49 100644 --- a/forum/CONCEPTS.md +++ b/forum/CONCEPTS.md @@ -1,54 +1,33 @@ -# Agent Forum v4 - Concepts & PoC Coverage Tracker +# Agent Forum v4 - Consolidated Master Tracker -This document tracks all concepts, data structures, and operational mechanisms -defined in the `agent-forum-v4` blueprint and `DATA_STRUCTURES.md`, along with -their Proof of Concept (PoC) coverage status. +This document serves as the single source of truth for all concepts, data structures, and operational mechanisms defined in the `agent-forum.md` architectural blueprint. It consolidates the high-level architecture with the granular implementation constraints previously split across satellite documentation. -## 1. Storage Layers (Git-Native Storage) +The table below tracks the definition, required implementation details, and Proof of Concept (PoC) coverage status for every implementable aspect of the Git-native ecosystem. -| Concept | Status | PoC File | Description | -| :--------------------------------------- | :----: | :--------------------- | :----------------------------------------------------------------------- | -| **1.0 Protocol Buffers (Protobuf)** | ✅ | `protobuf_poc.ts` | High-performance binary serialization for agent state transfer. | -| **1.1 Git Notes** | ✅ | `git_storage_poc.ts` | Attaching metadata/JSON to Git commits without altering file content. | -| **1.2 Orphan Branches (Meta-State)** | ✅ | `orphan_branch_poc.ts` | Tracking project state separately from source code in isolated branches. | -| **1.3 Embedded Vector DBs (sqlite-vec)** | ✅ | `vector_db_poc.ts` | Associative fuzzy memory retrieval of semantic data. | -| **1.4 TurboQuant** | ✅ | `vector_db_poc.ts` | Compressing vectors to binary hashes. Mocked alongside Vector DB PoC. | -| **1.5 Multi-Vec Isolation** | ✅ | `multi_vec_poc.ts` | Isolated sqlite-vec files preventing semantic bleed (e.g., docs vs telemetry). | -## 2. Process & Governance Structures -| Concept | Status | PoC File | Description | -| :------------------------------------------- | :----: | :--------------------- | :----------------------------------------------------------------- | -| **2.0 Declarative Frontmatter (YAML UUIDs)** | ✅ | `frontmatter_poc.ts` | Unique identification of artifacts via UUIDv7 in YAML frontmatter. | -| **2.1 The Project DAG (YAML)** | ✅ | `dag_engine_poc.ts` | Execution order driven by mathematical DAG resolution. | -| **2.2 Bounded Model Checking (Transitions)** | ✅ | `state_machine_poc.ts` | Strict state machine rules (transitions.json) for agent roles. | -| **2.3 The Constitution (AGENTS.md)** | ✅ | `constitution_poc.ts` | Strict adherence to the machine-readable ruleset/stack boundaries. | -| **2.4 Orchestration Matrix** | ✅ | `orchestration_matrix_poc.ts` | Definition and interactions of the 6 core agent roles (Gatekeeper, Historian, etc.). | - -## 3. Code Intelligence Structures - -| Concept | Status | PoC File | Description | -| :----------------------------------------------- | :----: | :------------------------- | :----------------------------------------------------------------- | -| **3.0 Git Merkle DAG Diffing** | ✅ | `merkle_diff_poc.ts` | O(1) context updates using native git tree diffing. | -| **3.1 SCIP Indexes (Semantic Code Protocol)** | ✅ | `code_intelligence_poc.ts` | Code symbol mapping (mocked via AST exports extraction). | -| **3.2 ASTs & Control Flow Graphs (CFGs)** | ✅ | `cfg_poc.ts` | Structured paths for security analysis (e.g., unsanitized inputs). | -| **3.3 Mutation Testing Scores** | ✅ | `mutation_poc.ts` | Blast radius / coverage enforcement by injecting bugs (Adversary). | -| **3.4 Dependency Graphing (Adjacency Matrices)** | ✅ | `dependency_graph_poc.ts` | Blast radius mapping via dependency matrices. | -| **3.5 Static Analysis Payloads** | ✅ | `static_analysis_poc.ts` | Ingestion of JSON outputs from tools like Semgrep/SonarQube. | - -## 4. Semantic & Telemetry Structures - -| Concept | Status | PoC File | Description | -| :----------------------------------------- | :----: | :----------------- | :-------------------------------------------------------------------- | -| **4.1 Ontologies (JSON-LD)** | ✅ | `ontology_poc.ts` | Deep traceability linking code/tasks to business requirements. | -| **4.2 OpenTelemetry Traces (.trace.json)** | ✅ | `telemetry_poc.ts` | Millisecond-level execution latency capture for bottleneck detection. | -| **4.3 Team Friction Telemetry** | ✅ | `telemetry_poc.ts` | Human-to-agent collaboration metrics (MTTR, Comment Ratios). | - -## 5. Architectural Graveyard (Dismissed Concepts) - -These concepts were explored but ultimately dismissed to respect strict repo-native constraints. See `GRAVEYARD.md` for details and rationales. - -| Concept | Status | Anti-PoC File | Description | -| :----------------------------------------- | :----: | :----------------- | :-------------------------------------------------------------------- | -| **5.1 Doc-to-LoRA Hypernetworks** | ❌ | `graveyard_poc.ts` | Dismissed due to extreme Git repository bloat from adapter weights. | -| **5.2 PASTE (Speculative Execution)** | ❌ | `graveyard_poc.ts` | Dismissed due to high risk of Bounded Model Checking violations. | +| Concept / Structure | Category | Blueprint Ref | Key Implementation Details | PoC File | Status | +| :--- | :--- | :--- | :--- | :--- | :--- | +| **Protocol Buffers (Protobuf)** | Data Structure | Section 1 | Serialized binary state and telemetry designed for conversion-less, high-performance data transfer. | `protobuf_poc.ts` | ✅ | +| **Git Notes** | Embedded Storage | Section 1 | Requires `git log --show-notes="forum/reasoning"` to fetch JSON payloads containing reasoning, risk, and telemetry. | `git_storage_poc.ts` | ✅ | +| **Orphan Branches** | Embedded Storage | Section 1 | Isolated `forum/meta-state` branch tracking CI/CD, RTMs, and DAGs without sharing main branch commit history. | `orphan_branch_poc.ts` | ✅ | +| **sqlite-vec & TurboQuant** | Embedded DB | Section 1 | 2-bit to 4-bit LSH/HNSW compression resulting in highly portable indexes typically under 30MB. | `vector_db_poc.ts` | ✅ | +| **Multi-Vec Isolation** | Embedded DB | Section 1 | Domain-specific SQLite files (e.g., `docs_graph.sqlite` distinct from `telemetry_graph.sqlite`) to prevent semantic bleed. | `multi_vec_poc.ts` | ✅ | +| **Declarative Frontmatter** | Data Structure | Section 1 & 4 | MUST utilize UUIDv7 (time-ordered) to allow historical sorting and chronological sequence inference directly from the ID. | `frontmatter_poc.ts` | ✅ | +| **The Project DAG** | Governance Structure | Section 4 | YAML schema explicitly requiring `id`, `blocked_by`, `legacy_slug`, `status`, and `description` fields. | `dag_engine_poc.ts` | ✅ | +| **Transitions Matrix** | Governance Structure | Section 1 & 2 | JSON mapping in `transitions.json` strictly defining Bounded Model Checking rules (e.g., `"Coder": { "requires": ["Gatekeeper_Approval"] }`). | `state_machine_poc.ts` | ✅ | +| **The Constitution (AGENTS.md)** | Governance Structure | Section 3 | Target application stack dynamically bound to the repository to statically prevent hallucinated dependencies. | `constitution_poc.ts` | ✅ | +| **Orchestration Matrix** | Governance Structure | Section 3 | Strict definition of inputs, outputs, and primary directives for the 6 core agents. | `orchestration_matrix_poc.ts` | ✅ | +| **Execution Flywheel** | Process | Section 5 | Continuous 3-step feedback loop: Analyst updates protocols, Gatekeeper reads constraints, cycle resets. | `execution_flywheel_poc.ts` | ⬜ | +| **Git Merkle DAG Diffing** | Data Structure | Section 2 | Relies on `git ls-tree` and `git diff-tree` hashes for zero-overhead, O(1) context updates. | `merkle_diff_poc.ts` | ✅ | +| **SCIP Indexes** | Data Structure | Section 2 | Lightweight database mapping code definitions and references, typically extracted via Tree-sitter. | `code_intelligence_poc.ts` | ✅ | +| **ASTs & CFGs** | Data Structure | Section 2 | JSON/XML mapping of variable execution paths utilized for deterministic security proofs. | `cfg_poc.ts` | ✅ | +| **Mutation & Adjacency** | Data Structure | Section 2 & 3 | Blast radius and coverage enforcement mappings generated by tools like Stryker and CodeSee. | `mutation_poc.ts` | ✅ | +| **Static Analysis Payloads** | Data Structure | Section 2 | Standardized JSON/XML metric outputs from industry tools like Semgrep and SonarQube. | `static_analysis_poc.ts` | ✅ | +| **Tool Sandbox** | Process | Appendix A | Safe local execution environment for CLI/WebAssembly binaries (Tree-sitter, Semgrep) respecting Bounded Model Checking constraints. | `tool_sandbox_poc.ts` | ⬜ | +| **Automated Git Hooks** | Process | Section 2 | Native pre-commit hooks configured to automatically generate SCIP and AST index graphs prior to agent interaction. | `git_hooks_poc.ts` | ⬜ | +| **Ontologies (JSON-LD)** | Data Structure | Section 4 | Embedded `@type: "Requirement"` blocks compiled to a mathematical `ontology.graph` file for deep traceability. | `ontology_poc.ts` | ✅ | +| **OTel Traces** | Data Structure | Section 4 | Standardized `.trace.json` payloads capturing millisecond-level execution latencies during testing. | `telemetry_poc.ts` | ✅ | +| **Team Friction Telemetry** | Data Structure | Section 4 | JSON payloads recorded in the meta-state branch tracking MTTR, PR comment-to-code ratios, and idle handoff durations. | `telemetry_poc.ts` | ✅ | +| **Doc-to-LoRA** | Dismissed Concept | Section 6 | Rejected because generating megabytes of `.safetensors` adapter weights directly causes extreme Git repository bloat. | `graveyard_poc.ts` | ❌ | +| **PASTE (Speculative Tooling)** | Dismissed Concept | Section 6 | Rejected because speculative write operations cause irrecoverable race conditions and corrupt the meta-state branch. | `graveyard_poc.ts` | ❌ | diff --git a/forum/DATA_STRUCTURES.md b/forum/DATA_STRUCTURES.md deleted file mode 100644 index 16ce7d4..0000000 --- a/forum/DATA_STRUCTURES.md +++ /dev/null @@ -1,175 +0,0 @@ -# Agent Forum v4 - Fundamental Data Structures Reference - -This document serves as the comprehensive list and reference for all data -structures, data sources, and embedded storage mechanisms outlined in the -`agent-forum-v4` blueprint. The goal is to provide a unified overview of the -machine-readable structures that agents will interact with, entirely eliminating -the need for external cloud SaaS databases. - -## 1. Storage Layers (Git-Native Storage) - -### 1.0 Protocol Buffers (Protobuf) - -- **Purpose**: Facilitates high-performance, conversion-less data transfer - between agents. -- **Content**: Serialized binary representations of agent state, telemetry, and - index data. -- **Integration**: Works natively with SCIP indexes and TurboQuant compressed - vector math to drastically reduce I/O latency. - -### 1.1 Git Notes (`refs/notes/commits`) - -- **Purpose**: Attaches arbitrary metadata directly to Git commits without - altering the commit hash or polluting the working directory. -- **Content**: Primarily JSON payloads containing: - - Agent "meta-thoughts" and reasoning. - - Risk assessments (e.g., generated by the Adversary). - - Telemetry summaries related to a specific commit. -- **Example Fetch**: `git log --show-notes="forum/reasoning"` - -### 1.2 Orphan Branches (Meta-State Branch) - -- **Purpose**: An isolated Git branch (e.g., `forum/meta-state`) that tracks - ongoing project state separately from the main source code. It shares no - commit history with the main branch. -- **Content**: - - CI/CD telemetry JSONs. - - Requirements Traceability Matrices (RTM). - - The Project Task DAG files. - - Periodic serialized graphs (e.g., SQLite dumps or graph snapshots). - -### 1.3 Embedded Vector Databases (`sqlite-vec`) - -- **Purpose**: Provides fuzzy, associative memory retrieval without a dedicated - network vector database. Compresses large documents via Locality-Sensitive - Hashing (LSH) and HNSW. -- **Content**: Highly compressed, quantized embeddings of concepts (PRDs, ADRs, - documentation). -- **Structure**: Isolated SQLite files (e.g., `docs_graph.sqlite`, - `telemetry_graph.sqlite`) to prevent cross-contamination of semantic data. - -### 1.4 TurboQuant - -- **Purpose**: Compresses high-dimensional semantic concepts into binary hashes - using 2-bit to 4-bit quantization. -- **Content**: Extremely lightweight local embedded indexes (often under 30MB) - facilitating millisecond vector search inside `sqlite-vec`. - -### 1.5 Multi-Vec Isolation - -- **Purpose**: Prevents semantic bleed by utilizing isolated, separate `sqlite-vec` files rather than dumping all embeddings into a single vector database. -- **Content**: Domain-specific embedding files (e.g., `docs_graph.sqlite` distinct from `telemetry_graph.sqlite`). - -## 2. Process & Governance Structures - -### 2.0 Declarative Frontmatter (YAML UUIDs) - -- **Purpose**: Uniquely identifies Markdown artifacts to maintain traceability - within the project DAG and the vector databases. -- **Content**: YAML blocks containing a unique UUID (Artifact-ID). -- **Format Note**: MUST be compatible with UUIDv7 (time-ordered) to allow - historical sorting and chronological sequence inference directly from the - identifier, acting as a strict primary key. - -### 2.1 The Project DAG (YAML) - -- **Purpose**: Replaces traditional flat project management tools (like Jira or - Markdown task lists). Dictates execution order mathematically. -- **Content**: YAML files representing a Directed Acyclic Graph. -- **Key Fields**: - - `id`: A UUIDv7 acting as the unique identifier. - - `blocked_by`: Array of UUIDs this task depends on. - - `legacy_slug` (Optional): The visual human-readable string (e.g., - `YYYY-MMDD.[sequence]...`). - - `status`: e.g., `pending`, `in_progress`, `completed`. - - `description`: The actual prompt/goal. - -### 2.2 Bounded Model Checking (Transitions Matrix) - -- **Purpose**: Defines strict state machine rules for the agent pipeline to - guarantee proper governance. -- **Format**: `transitions.json` -- **Content**: A JSON mapping that states which roles can execute under which - conditions (e.g., `"Coder": { "requires": ["Gatekeeper_Approval"] }`). - -### 2.3 The Constitution (`AGENTS.md`) - -- **Purpose**: The supreme machine-readable ruleset that all agents must ingest - to understand the target application stack, constraints, and operational - boundaries. - -### 2.4 The Orchestration Matrix (Agent Roles) - -- **Purpose**: Defines the inputs, outputs, and primary directives of the 6 core agents. -- **Roles**: - - **Gatekeeper**: Translates Ontologies/DAGs to Verification Checklists. - - **Historian**: Uses sqlite-vec and Git Notes to inject historical context. - - **Adversary**: Reads SCIP, CFGs, and Mutation data to generate tests/mutations. - - **Translator**: Consumes SCIP diffs to output API references and docs. - - **Analyst**: Reads Telemetry to propose workflow optimizations. - - **Evaluator**: Governs pipeline progression by reading transitions.json. - -## 3. Code Intelligence Structures - -### 3.0 Git Merkle DAG Diffing - -- **Purpose**: Ensures O(1) context updates for agents by identifying exact - modified file hashes without reading raw file strings. -- **Content**: Hashes resulting from zero-overhead diffing (e.g., `git ls-tree` - and `git diff-tree`). - -### 3.1 SCIP Indexes (Semantic Code Intelligence Protocol) - -- **Purpose**: Replaces unreliable regex-based searching with a statically - guaranteed mapping of code symbols. -- **Content**: A lightweight database mapping definitions, references, and - relationships across the codebase. Extracted typically via Tree-sitter. - -### 3.2 Abstract Syntax Trees (ASTs) & Control Flow Graphs (CFGs) - -- **Purpose**: Structured representations of code syntax and execution paths. -- **Content**: JSON/XML mapping of every possible path a variable can take, used - by the Adversary agent to deterministically prove security flaws (e.g., - unsanitized inputs reaching SQL statements). - -### 3.3 Mutation Testing Scores - -- **Purpose**: Represents the "blast radius" and effectiveness of test suites. -- **Content**: Structured outputs from tools like Stryker or Mutmut that - indicate how many injected bugs were successfully caught by the test physics. - -### 3.4 Dependency Graphing (Adjacency Matrices) - -- **Purpose**: Mathematically calculates the exact "blast radius" of any code - change. -- **Content**: Adjacency matrices (generated by tools like CodeSee or Madge) - that map the downstream and upstream impact across components. - -### 3.5 Static Analysis Payloads - -- **Purpose**: Provides compiler-grade code smell and vulnerability metrics directly to triage agents. -- **Content**: Standardized JSON/XML outputs from industry tools (e.g., Semgrep, SonarQube). - -## 4. Semantic & Telemetry Structures - -### 4.1 Ontologies (JSON-LD) - -- **Purpose**: Replaces legacy requirements management (like DOORS). Achieves - deep traceability by linking code/tasks to business requirements. -- **Content**: Linked Data JSON blocks embedded in documentation - (`@type: "Requirement"`). These compile into a single mathematical - `ontology.graph` file. - -### 4.2 OpenTelemetry Traces (`.trace.json`) - -- **Purpose**: Captures millisecond-level execution latencies during testing. -- **Content**: Standardized OTel trace JSON payloads ingested by the Adversary - to find physical execution bottlenecks in the code. - -### 4.3 Team Friction Telemetry - -- **Purpose**: Used by the Analyst agent to measure the efficiency of - human-to-agent collaboration. -- **Content**: JSON payloads stored in the meta-state branch recording metrics - like Mean Time to Resolution (MTTR), PR comment-to-code ratios, and idle - handoff durations. diff --git a/forum/GRAVEYARD.md b/forum/GRAVEYARD.md deleted file mode 100644 index 12ab721..0000000 --- a/forum/GRAVEYARD.md +++ /dev/null @@ -1,29 +0,0 @@ -# Architectural Graveyard - -This document catalogs advanced technical approaches and bleeding-edge tools that were evaluated during the design phase of `agent-forum` but were ultimately dismissed. - -The primary purpose of this file is to mathematically or logically prove *why* these concepts violate the strict constraints of a Git-Native, Local-First ecosystem. We document these anti-patterns to prevent future regressions where an agent or engineer might attempt to re-introduce them without understanding the structural consequences. - -## 1. Doc-to-LoRA (D2L) Hypernetworks - -**The Concept:** -A Perceiver-based latent mapping system designed to internalize external context by generating LoRA (Low-Rank Adaptation) weights in a single forward pass. This eliminates KV-cache overhead during inference by directly modifying the neural network's weights based on project documentation. - -**The Dismissal Rationale (Git Repository Bloat):** -While D2L offers incredible inference speeds, the resulting `.safetensors` adapter files (even low-rank ones) are typically several megabytes in size. -In a Git-Native ecosystem, state must be tracked alongside the code. If an agent generates a new LoRA adapter for every major architectural change or task, committing thousands of binary `.safetensors` files directly to the Git object database will cause extreme repository bloat, violating the goal of a lightweight, portable codebase. - -**The Alternative:** -D2L was swapped out in favor of context-caching via `sqlite-vec` (Embedded Vector Databases). High-dimensional semantic data is compressed via TurboQuant (2-bit to 4-bit quantization) into binary hashes, resulting in a tiny, highly portable index (often under 30MB) that can be queried natively in milliseconds. - -## 2. PASTE (Pattern-Aware Speculative Tool Execution) - -**The Concept:** -A framework that predicts tool calls using historical patterns and executes them speculatively *while* the LLM is still generating text. This parallel execution aims to achieve near-zero latency by having the tool results ready the moment the LLM decides it needs them. - -**The Dismissal Rationale (Bounded Model Checking Violations):** -The `agent-forum` architecture relies heavily on strict mathematical governance (Bounded Model Checking via `transitions.json`). Agents are restricted to a defined Orchestration Matrix. -Speculative execution fundamentally breaks this governance. If an agent speculatively executes a script (e.g., to read a file, or modify a meta-state JSON) before the Gatekeeper or Evaluator has authorized the transition, it violates the deterministic, step-by-step state machine. Furthermore, speculative write operations (e.g., speculatively drafting a commit) can lead to irrecoverable race conditions and corrupted meta-state branches if the LLM ultimately decides *not* to use the tool. - -**The Alternative:** -Execution must remain purely deterministic. Tool execution happens strictly sequentially, driven by the DAG and the Transitions Matrix. Speed is achieved not through speculative branching, but through O(1) Merkle DAG diffing and zero-latency local vector lookups, keeping the input context tiny and execution times low. diff --git a/forum/agent-forum-v4.md b/forum/agent-forum-v4.md deleted file mode 100644 index 395580d..0000000 --- a/forum/agent-forum-v4.md +++ /dev/null @@ -1,226 +0,0 @@ -# **Agent Forum** - -## **Git-Native Agent Collaboration Ecosystem** - -This galactic report defines the architectural blueprint for a Git-native, -hyper-efficient AI agent ecosystem. By constraining all state, memory, and -tooling to the local repository, cloud SaaS dependencies are replaced with -embedded data structures (Merkle DAGs, SCIP indexes, local vector graphs, YAML -task DAGs). This creates a zero-latency, cryptographically immutable pipeline -where AI agents interact with structural code physics and semantic ontologies -rather than raw text. - -## **1\. The Git-Native Forcing Function & Embedded Storage** - -Constraining the state and tooling entirely within the repository format acts as -a brilliant forcing function. It shifts the architecture from a "Cloud-Native" -distributed system to a "Local-First / Git-Native" operating system. Removing -third-party databases preserves project isolation and provides cryptographic -immutability with zero-latency access. - -- **Memory Storage (Git Notes & Orphan Branches):** - - **Git Notes (refs/notes/commits):** Arbitrary metadata—such as JSON - transcripts of an AI agent's decision-making process—is attached directly to - a commit without altering the commit hash. The Historian agent can read git - log \--show-notes="ai" to understand why a specific line of code was - written, keeping the working directory clean. - - **The Meta-State Orphan Branch:** Ongoing project state, such as CI/CD - telemetry and Requirements Traceability Matrices (RTM), is tracked in a - parallel orphan branch. Agents commit dynamic state JSONs here, isolated - within the same .git folder but completely separate from the main source - code. - - **Serialization (JSON vs. Protocol Buffers):** While JSON is utilized for - human-readable state tracking, engineering teams should evaluate Protocol - Buffers (Protobuf) for high-performance, conversion-less data transfer - between agents. Protobuf integrates natively with SCIP indexes and works in - tandem with TurboQuant (which compresses the vector math), drastically - reducing I/O latency.\ - **Impact:** Eliminates reliance on external databases while maintaining - perfect, version-controlled state isolation. -- **Fuzzy Retrieval via Embedded Vector Search:** - - **sqlite-vec & TurboQuant:** Traditional databases require exact keyword - matches, but Locality-Sensitive Hashing (LSH) and Hierarchical Navigable - Small World (HNSW) algorithms compress high-dimensional concepts into binary - hashes. Using the sqlite-vec extension with 2-bit to 4-bit "TurboQuant" - quantization allows massive semantic knowledge (PRDs, ADRs) to be compressed - into a tiny local file (often under 30MB). Agents can query these - associative memories in milliseconds without network calls. - - **Multi-Vec Isolation:** Rather than dumping all embeddings into a single - vector database, the meta-state branch should consider isolated sqlite-vec - files (e.g., docs\_graph.sqlite and telemetry\_graph.sqlite). This - "Multi-Vec" architecture prevents semantic bleed, ensuring a query about - code performance does not cross-contaminate with team communication logs.\ - **Impact:** Reduces context window bloat and eliminates cloud database - latency. -- **Protocols & Governance:** - - **Declarative Frontmatter:** Every Markdown artifact requires YAML - frontmatter containing a unique UUID (Artifact-ID). - - **Bounded Model Checking (BMC):** A local state machine reads a static - .agents/transitions.json file to dictate the execution pipeline. This - ensures strict governance (e.g., "The Coder agent cannot run until the - Gatekeeper agent has signed off"). - -## **2\. Structured Code Intelligence** - -To prevent context window collapse and massive compute costs, agents must not -ingest raw text. Instead, they require a highly efficient I/O pipeline built on -structured code intelligence. - -- **Git Merkle DAG Diffing:** Because Git is fundamentally a Merkle Tree, the - system uses zero-overhead diffing (git ls-tree and git diff-tree) to instantly - identify changed file hashes. The AI's knowledge base updates in milliseconds - by walking down the tree to the exact modified file.\ - **Impact:** Guarantees O(1) context updates by passing only cryptographic - diffs rather than full file strings. -- **From Syntax to Code Property Graphs (CPGs):** - - **Tree-sitter & SCIP Indexes:** Instead of regex, Tree-sitter incrementally - parses code into a structured Abstract Syntax Tree (AST). A pre-commit hook - then generates a SCIP (Semantic Code Intelligence Protocol) index—a - lightweight database of code symbols providing statically guaranteed "Find - References" and "Go to Definition" capabilities. - - **Control Flow Graphs (CFGs):** Extracted from the AST, CFGs map every - possible path a variable can take. The Adversary agent can feed this JSON - dataset into its prompt to deterministically prove if unsanitized user input - can ever reach a database query.\ - **Impact:** Transforms ambiguous text processing into deterministic, - mathematically verifiable graph traversals. -- **Human-Grade Quality Tools:** Agents ingest the JSON/XML outputs of - industry-standard tools: - - **Static Analysis (Semgrep / SonarQube):** Feeds vulnerabilities and code - smells directly to triage agents. - - **Mutation Testing (Stryker / Mutmut):** Injects bugs to test the tests. - Feeding mutation scores to the Adversary agent forces the generation of - edge-case coverage rather than superficial line-coverage. - - **Dependency Graphing (CodeSee / Madge):** Generates adjacency matrices to - calculate the exact "blast radius" of a code change.\ - **Impact:** Roots agent decision-making in industry-standard, compiler-grade - telemetry rather than LLM guesswork. - -## **3\. Orchestration Matrix & Governance** - -The AGENTS.md file serves as the strict, machine-readable constitution. To -ensure agent autonomy, instructions must rely on this repository documentation -rather than micromanaging or spoon-feeding step-by-step logic in individual -system prompts. - -| Role | Inputs | Outputs | Primary Directive | -| :------------- | :--------------------------------------- | :--------------------------------------- | :-------------------------------------------------------------------------------- | -| **Gatekeeper** | Ontologies, YAML DAGs | Verification checklists | Bridge human requirements with technical reality. | -| **Historian** | sqlite-vec, Git Notes | Contextual injection | Prevent regression and historical repetition. | -| **Adversary** | SCIP graphs, CFGs, Mutation, OTel Traces | Edge-case tests, mutations, bottlenecks | Expose security flaws, enforce test coverage, and identify execution bottlenecks. | -| **Translator** | SCIP diffs, existing docs | API references, guides | Maintain code-to-documentation parity. | -| **Analyst** | Telemetry, PR threads | Workflow optimizations, Protocol updates | Optimize human-to-agent collaboration. | -| **Evaluator** | transitions.json, DAGs | Pipeline progression | Govern pipeline integrity (R/W access to meta-state). | - -### **The Adversary's Expanded Scope** - -Traditionally associated solely with security, this agent wears three distinct -hats to comprehensively stress-test the repository: - -1. **The Security Auditor:** Feeds on Control Flow Graphs (CFGs) to - deterministically prove if unsanitized user input reaches database queries. -2. **The Quality Engineer:** Consumes mutation scores (from Stryker/Mutmut) to - hunt for edge cases and enforce strict test coverage. -3. **The Performance Engineer:** Ingests OpenTelemetry .trace.json files from - Section 4 to identify real-world execution bottlenecks. - -### **Target Application Stack Boundaries** - -All agents must adhere to the defined stack: Native Web Components with -Declarative Shadow DOM, Vue.js, Pinia, Tailwind CSS, Deno (TypeScript), Rust for -local systems, and PostgreSQL. - -_Implementation Note:_ The Vue.js/Deno/Rust stack listed above is a strictly -defined example. The core architectural rule is that the target tech stack must -be explicitly and exhaustively defined in the AGENTS.md file. By locking in the -stack, agents are statically prevented from hallucinating unauthorized -libraries, frameworks, or legacy dependencies into the codebase. - -## **4\. Semantic Project Management & Telemetry** - -By mapping the syntactic structure of code to the semantic structure of a -project, the system establishes concrete datasets that act as the connective -tissue between code, schedules, and business logic. - -- **Replacing Jira (The Project DAG):** Project stories are serialized into the - meta-state branch as strict YAML DAGs (e.g., Task\_44 explicitly declares - blocked\_by: \[Task\_42, Task\_43\]). On every commit, the Evaluator agent - reads the DAG to calculate the critical path, unblocking tasks and preventing - agents from executing code out of order. -- **Replacing DOORS (The Ontology):** Deep traceability is achieved by embedding - JSON-LD (Linked Data) blocks at the top of markdown documents (@type: - "Requirement"). A script compiles these into a single ontology.graph file. - Agents query this graph mathematically to find all components with - relationship edges to specific business requirements. -- **Execution Traces (The Physics):** OpenTelemetry (OTel) traces are generated - during test runs as .trace.json files, capturing millisecond execution - latency. The Adversary agent uses this to understand how the code actually - runs, identifying bottlenecks with precision. -- **Communication Telemetry:** The Analyst consumes specific metrics—Mean Time - to Resolution (MTTR), PR Comment-to-Code Ratio, Idle Handoff Duration, - Artifact Override Frequency, and Thread Friction Markers—serialized as JSON - payloads in the meta-state branch to map team friction. - -## **5\. The Execution Pipeline** - -The entire system operates as a continuous, structured data flywheel. All -artifacts are embedded into the local database, providing agents with a perfect, -multi-dimensional understanding of the repository. - -1. Analyst interprets telemetry to update project protocols. -2. Gatekeeper reads these new protocols to constrain the next cycle. -3. The loop resets, returning to code generation with updated guardrails. - -| | Artifact | Generated Data Structure | Primary Consumer Role | -| :------ | :---------------------------- | :----------------------- | :--------------------- | -| **1\.** | **The Code** (Architecture) | SCIP/ASTs | Adversary / Translator | -| **2\.** | **The Tests** (Physics) | OTel Traces | Adversary | -| **3\.** | **The Docs** (Business Logic) | JSON-LD Ontologies | Gatekeeper | -| **4\.** | **The Process** (Schedule) | YAML DAGs | Evaluator | -| **5\.** | **The Team** (Friction) | JSON Telemetry | Analyst | - -This pipeline is not a linear checklist; it is a continuous, self-correcting -feedback loop. As demonstrated above, Step 5 (The Team generates Telemetry) -feeds directly back into Step 1\. When the Analyst identifies workflow friction, -it updates project protocols, which directly dictates how the Gatekeeper sets -constraints for the next cycle of Code generation. - -## **6\. Filtered Explorations (Architectural Graveyard)** - -During the design phase, several bleeding-edge tools were evaluated but -ultimately altered to respect the strict repo-native constraints. - -- **Doc-to-LoRA (D2L) Hypernetworks:** A Perceiver-based latent mapping system - designed to internalize external context by generating LoRA weights in a - single forward pass, eliminating KV-cache overhead. - - _The Verdict:_ While incredibly fast for inference, committing thousands of - `.safetensors` adapter weights to Git would inevitably bloat the repository. - D2L was swapped out in favor of context-caching via `sqlite-vec`. -- **PASTE (Pattern-Aware Speculative Tool Execution):** A framework that - predicts tool calls using historical patterns and executes them while the LLM - is still generating to achieve near-zero latency. - - _The Verdict:_ Highly valuable for meta-routing, but its implementation - requires careful tuning to ensure speculative executions do not violate the - local computing and Bounded Model Checking constraints of the repository - graph. - -## **Appendix A: Example Toolchain Catalog** - -To extract the structured data required by the AI agents, the following external -and custom utilities constitute some example utilities. Engineering teams should -expand this catalog as specific disciplinary data sets are required or -perfected. Custom options are ok if they provide valuable benefit. - -- **Syntax & Architecture (SCIP/AST Extraction):** Tree-sitter (Local - WebAssembly binaries for generating Abstract Syntax Trees) and SCIP CLI - (Generates the Semantic Code Intelligence Protocol graphs). -- **Security & Static Analysis:** Semgrep / SonarQube (Compiles vulnerabilities - and code smells into JSON payloads for the Adversary). -- **Quality & Mutation Testing:** Stryker / Mutmut (Injects bugs during the CI - cycle to generate edge-case mutation scores). -- **Physics & Telemetry:** OpenTelemetry / OTel (Extracts millisecond execution - latency into `.trace.json` files). -- **Dependency & Blast Radius:** CodeSee / Madge (Generates adjacency matrices - to map downstream impact of code changes). -- **Data Storage & Retrieval:** sqlite-vec (Embedded SQLite extensions handling - local vector indexing and TurboQuant compression). diff --git a/forum/agent-forum.md b/forum/agent-forum.md new file mode 100644 index 0000000..2d18090 --- /dev/null +++ b/forum/agent-forum.md @@ -0,0 +1,112 @@ +# **Agent Forum \- v6** + +## **Git-Native Agent Collaboration Ecosystem** + +This galactic report defines the architectural blueprint for a Git-native, hyper-efficient AI agent ecosystem. By constraining all state, memory, and tooling to the local repository, cloud SaaS dependencies are replaced with embedded data structures (Merkle DAGs, SCIP indexes, local vector graphs, YAML task DAGs). This creates a zero-latency, cryptographically immutable pipeline where AI agents interact with structural code physics and semantic ontologies rather than raw text. + +## **1\. The Git-Native Forcing Function & Embedded Storage** + +Constraining the state and tooling entirely within the repository format acts as a brilliant forcing function. It shifts the architecture from a "Cloud-Native" distributed system to a "Local-First / Git-Native" operating system. Removing third-party databases preserves project isolation and provides cryptographic immutability with zero-latency access. + +* **Memory Storage (Git Notes & Orphan Branches):** + * **Git Notes (refs/notes/commits):** Arbitrary metadata—such as JSON transcripts of an AI agent's decision-making process—is attached directly to a commit without altering the commit hash. The Historian agent can read git log \--show-notes="ai" to understand why a specific line of code was written, keeping the working directory clean. + * **The Meta-State Orphan Branch:** Ongoing project state, such as CI/CD telemetry and Requirements Traceability Matrices (RTM), is tracked in a parallel orphan branch. Agents commit dynamic state JSONs here, isolated within the same .git folder but completely separate from the main source code. + * **Serialization (JSON vs. Protocol Buffers):** While JSON is utilized for human-readable state tracking, engineering teams should evaluate Protocol Buffers (Protobuf) for high-performance, conversion-less data transfer between agents. Protobuf integrates natively with SCIP indexes and works in tandem with TurboQuant (which compresses the vector math), drastically reducing I/O latency. + **Impact:** Eliminates reliance on external databases while maintaining perfect, version-controlled state isolation. +* **Fuzzy Retrieval via Embedded Vector Search:** + * **sqlite-vec & TurboQuant:** Traditional databases require exact keyword matches, but Locality-Sensitive Hashing (LSH) and Hierarchical Navigable Small World (HNSW) algorithms compress high-dimensional concepts into binary hashes. Using the sqlite-vec extension with 2-bit to 4-bit "TurboQuant" quantization allows massive semantic knowledge (PRDs, ADRs) to be compressed into a tiny local file (often under 30MB). Agents can query these associative memories in milliseconds without network calls. + * **Multi-Vec Isolation:** Rather than dumping all embeddings into a single vector database, the meta-state branch should consider isolated sqlite-vec files (e.g., docs\_graph.sqlite and telemetry\_graph.sqlite). This "Multi-Vec" architecture prevents semantic bleed, ensuring a query about code performance does not cross-contaminate with team communication logs. + **Impact:** Reduces context window bloat and eliminates cloud database latency. +* **Protocols & Governance:** + * **Declarative Frontmatter:** Every Markdown artifact requires YAML frontmatter containing a unique UUID (Artifact-ID). + * **Bounded Model Checking (BMC):** A local state machine reads a static .agents/transitions.json file to dictate the execution pipeline. This ensures strict governance (e.g., "The Coder agent cannot run until the Gatekeeper agent has signed off"). + +## **2\. Structured Code Intelligence** + +To prevent context window collapse and massive compute costs, agents must not ingest raw text. Instead, they require a highly efficient I/O pipeline built on structured code intelligence. + +* **Git Merkle DAG Diffing:** Because Git is fundamentally a Merkle Tree, the system uses zero-overhead diffing (git ls-tree and git diff-tree) to instantly identify changed file hashes. The AI's knowledge base updates in milliseconds by walking down the tree to the exact modified file. + **Impact:** Guarantees O(1) context updates by passing only cryptographic diffs rather than full file strings. +* **From Syntax to Code Property Graphs (CPGs):** + * **Tree-sitter & SCIP Indexes:** Instead of regex, Tree-sitter incrementally parses code into a structured Abstract Syntax Tree (AST). A pre-commit hook then generates a SCIP (Semantic Code Intelligence Protocol) index—a lightweight database of code symbols providing statically guaranteed "Find References" and "Go to Definition" capabilities. + * **Control Flow Graphs (CFGs):** Extracted from the AST, CFGs map every possible path a variable can take. The Adversary agent can feed this JSON dataset into its prompt to deterministically prove if unsanitized user input can ever reach a database query. + **Impact:** Transforms ambiguous text processing into deterministic, mathematically verifiable graph traversals. +* **Human-Grade Quality Tools:** Agents ingest the JSON/XML outputs of industry-standard tools: + * **Static Analysis (Semgrep / SonarQube):** Feeds vulnerabilities and code smells directly to triage agents. + * **Mutation Testing (Stryker / Mutmut):** Injects bugs to test the tests. Feeding mutation scores to the Adversary agent forces the generation of edge-case coverage rather than superficial line-coverage. + * **Dependency Graphing (CodeSee / Madge):** Generates adjacency matrices to calculate the exact "blast radius" of a code change. + **Impact:** Roots agent decision-making in industry-standard, compiler-grade telemetry rather than LLM guesswork. + +## **3\. Orchestration Matrix & Governance** + +The AGENTS.md file serves as the strict, machine-readable constitution. To ensure agent autonomy, instructions must rely on this repository documentation rather than micromanaging or spoon-feeding step-by-step logic in individual system prompts. + +| Role | Inputs | Outputs | Primary Directive | +| :---- | :---- | :---- | :---- | +| **Gatekeeper** | Ontologies, YAML DAGs | Verification checklists | Bridge human requirements with technical reality. | +| **Historian** | sqlite-vec, Git Notes | Contextual injection | Prevent regression and historical repetition. | +| **Adversary** | SCIP graphs, CFGs, Mutation, OTel Traces | Edge-case tests, mutations, bottlenecks | Expose security flaws, enforce test coverage, and identify execution bottlenecks. | +| **Translator** | SCIP diffs, existing docs | API references, guides | Maintain code-to-documentation parity. | +| **Analyst** | Telemetry, PR threads | Workflow optimizations, Protocol updates | Optimize human-to-agent collaboration. | +| **Evaluator** | transitions.json, DAGs | Pipeline progression | Govern pipeline integrity (R/W access to meta-state). | + +### **The Adversary's Expanded Scope** + +Traditionally associated solely with security, this agent wears three distinct hats to comprehensively stress-test the repository: + +1. **The Security Auditor:** Feeds on Control Flow Graphs (CFGs) to deterministically prove if unsanitized user input reaches database queries. +2. **The Quality Engineer:** Consumes mutation scores (from Stryker/Mutmut) to hunt for edge cases and enforce strict test coverage. +3. **The Performance Engineer:** Ingests OpenTelemetry .trace.json files from Section 4 to identify real-world execution bottlenecks. + +### **Target Application Stack Boundaries** + +The operational technology stack is strictly and dynamically defined by the repository's AGENTS.md (The Constitution). Agents are mathematically bound to the stack declared in this file. By locking in the stack at the repository level, agents are statically prevented from hallucinating unauthorized libraries, frameworks, legacy dependencies, or unapproved languages into the codebase. + +## **4\. Semantic Project Management & Telemetry** + +By mapping the syntactic structure of code to the semantic structure of a project, the system establishes concrete datasets that act as the connective tissue between code, schedules, and business logic. + +* **Replacing Jira (The Project DAG):** Project stories are serialized into the meta-state branch as strict YAML DAGs (e.g., Task\_44 explicitly declares blocked\_by: \[Task\_42, Task\_43\]). On every commit, the Evaluator agent reads the DAG to calculate the critical path, unblocking tasks and preventing agents from executing code out of order. +* **Replacing DOORS (The Ontology):** Deep traceability is achieved by embedding JSON-LD (Linked Data) blocks at the top of markdown documents (@type: "Requirement"). A script compiles these into a single ontology.graph file. Agents query this graph mathematically to find all components with relationship edges to specific business requirements. +* **Execution Traces (The Physics):** OpenTelemetry (OTel) traces are generated during test runs as .trace.json files, capturing millisecond execution latency. The Adversary agent uses this to understand how the code actually runs, identifying bottlenecks with precision. +* **Communication Telemetry:** The Analyst consumes specific metrics—Mean Time to Resolution (MTTR), PR Comment-to-Code Ratio, Idle Handoff Duration, Artifact Override Frequency, and Thread Friction Markers—serialized as JSON payloads in the meta-state branch to map team friction. + +## **5\. The Execution Pipeline** + +The entire system operates as a continuous, structured data flywheel. All artifacts are embedded into the local database, providing agents with a perfect, multi-dimensional understanding of the repository. + +To visualize this flow, the five core stages of the pipeline map directly to the artifacts they generate and the specific roles that consume them: + +| | Artifact | Generated Data Structure | Primary Consumer Role | +| :---- | :---- | :---- | :---- | +| **1\.** | **The Code** (Architecture) | SCIP/ASTs | Adversary / Translator | +| **2\.** | **The Tests** (Physics) | OTel Traces | Adversary | +| **3\.** | **The Docs** (Business Logic) | JSON-LD Ontologies | Gatekeeper | +| **4\.** | **The Process** (Schedule) | YAML DAGs | Evaluator | +| **5\.** | **The Team** (Friction) | JSON Telemetry | Analyst | + +This pipeline is not a linear checklist; it is a continuous, self-correcting feedback loop. As demonstrated above, Step 5 (The Team generates Telemetry) feeds directly back into Step 1 to optimize the next pass: + +1. The **Analyst** interprets telemetry to update project protocols. +2. The **Gatekeeper** reads these new protocols to constrain the next cycle. +3. The loop resets, returning to code generation with updated guardrails. + +## **6\. Filtered Explorations (Architectural Graveyard)** + +During the design phase, several bleeding-edge tools were evaluated but ultimately altered to respect the strict repo-native constraints. + +* **Doc-to-LoRA (D2L) Hypernetworks:** A Perceiver-based latent mapping system designed to internalize external context by generating LoRA weights in a single forward pass, eliminating KV-cache overhead. + * *The Verdict:* While incredibly fast for inference, committing thousands of .safetensors adapter weights to Git would inevitably bloat the repository. D2L was swapped out in favor of context-caching via sqlite-vec. +* **PASTE (Pattern-Aware Speculative Tool Execution):** A framework that predicts tool calls using historical patterns and executes them while the LLM is still generating to achieve near-zero latency. + * *The Verdict:* Highly valuable for meta-routing, but its implementation requires careful tuning to ensure speculative executions do not violate the local computing and Bounded Model Checking constraints of the repository graph. + +## **Appendix A: Example Toolchain Catalog** + +To extract the structured data required by the AI agents, the following external utilities serve as strong baseline candidates. Engineering teams are explicitly encouraged to expand or substitute this catalog as new tools emerge or specific disciplinary datasets are required. While the underlying data structure requirements are strictly governed, tool choice remains highly flexible—custom integrations and alternative products are welcome provided they satisfy the deterministic extraction goals of the pipeline. + +* **Syntax & Architecture (SCIP/AST Extraction):** Tree-sitter (Local WebAssembly binaries for generating Abstract Syntax Trees) and SCIP CLI (Generates the Semantic Code Intelligence Protocol graphs). +* **Security & Static Analysis:** Semgrep / SonarQube (Compiles vulnerabilities and code smells into JSON payloads for the Adversary). +* **Quality & Mutation Testing:** Stryker / Mutmut (Injects bugs during the CI cycle to generate edge-case mutation scores). +* **Physics & Telemetry:** OpenTelemetry / OTel (Extracts millisecond execution latency into .trace.json files). +* **Dependency & Blast Radius:** CodeSee / Madge (Generates adjacency matrices to map downstream impact of code changes). +* **Data Storage & Retrieval:** sqlite-vec (Embedded SQLite extensions handling local vector indexing and TurboQuant compression). \ No newline at end of file