Merge pull request #16 from mrteye/feat-audit-merkle-ledger-task-5295491276490979898

feat: Add Merkle Tree Ledger task file

tasks/new/2026-0824.01.jul.feat.audit-merkle.add-merkle-tree-ledger-0547.md

@@ -0,0 +1,117 @@
+# TASK METADATA
+
+- **Target Files:** `server/audit.ts`, `server/audit_merkle.ts`,
+  `server/valkey.ts`, `server/db.ts`, and database schema/migration files.
+- **Core Objective:** Implement an append-only RFC 6962 Merkle Tree audit ledger
+  in PostgreSQL with an in-process Deno micro-batcher for STH computation and
+  Valkey pub/sub broadcast.
+- **Dependencies:** Deno 2.x WebCrypto (Ed25519), Valkey connection, PostgreSQL
+  schema updates, and SPIFFE workload identity key material.
+- **Additional Important Notes:** Must use Deno's native WebCrypto (no external
+  crypto npm packages). Valkey session keys should use raw strings without
+  'session:' prefix (not fully applicable here, but good standard). Ensure tests
+  mock Valkey, PostgreSQL, and SPIFFE cleanly.
+
+---
+
+## Architectural Considerations & Risks
+
+Before implementing the RFC 6962 Merkle Tree Ledger, we must evaluate the
+current architecture and explicitly identify risks, regressions, and necessary
+constraints.
+
+**Risks & Tradeoffs:**
+
+1. **In-Process Micro-Batcher Memory & State:** The micro-batcher runs
+   in-process inside Deno using asynchronous intervals (e.g., `setInterval` or
+   explicit flush). If the Deno process crashes abruptly, unflushed audit logs
+   in memory could be lost before the batch is committed to PostgreSQL and the
+   Merkle tree head is updated.
+   - _Mitigation:_ Ensure `auditLog` still performs an immediate async insert of
+     the _record_ to the database (as it currently does via `server/audit.ts`),
+     but the `leaf_hash` and tree updates are computed dynamically or buffered
+     for the batched STH computation. The database remains the source of truth;
+     the batcher just finalizes the cryptographical proof.
+2. **Database Schema Locking:** Adding a `leaf_hash` column to the
+   `audit_records` table and creating the `audit_sths` table might cause locking
+   if the table is large.
+   - _Mitigation:_ The `leaf_hash` can be computed prior to insert.
+3. **SPIFFE Identity Dependency:** STH generation requires the Deno process to
+   sign using the SPIFFE workload identity/server signing key via WebCrypto. If
+   SPIFFE/FFI fails or the keys are rotating during a batch, the STH generation
+   might fail.
+   - _Mitigation:_ Graceful error handling and retry mechanism for STH signing.
+4. **No Third-Party Dependencies:** Adhering to the memory context, we must
+   exclusively use Deno 2.x native `crypto.subtle` API for SHA-256 and
+   Ed25519/ECDSA, rather than any Node.js polyfills or npm packages.
+
+**Alternatives Evaluated:**
+
+- _External Worker:_ Using a dedicated worker (e.g., a separate service or CRON)
+  for computing the Merkle root. _Decision:_ We rejected this to keep the
+  application self-contained and zero-dependency, per user guidance.
+- _Synchronous Tree Updates:_ Recomputing the tree on every single write.
+  _Decision:_ Rejected. This would severely block relational write throughput. A
+  micro-batcher (e.g., 30-60s or size threshold) is the optimal path for
+  decoupling heavy crypto logic from the request hot-path.
+
+## Proposed Implementation
+
+The execution of this feature will be broken down into the following structured
+phases:
+
+### Phase 1: Database Schema Expansion
+
+1. Update PostgreSQL schema to alter `audit_records` and add a `leaf_hash TEXT`
+   column.
+2. Create the new `audit_sths` table:
+   ```sql
+   CREATE TABLE audit_sths (
+     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+     tree_size BIGINT NOT NULL,
+     root_hash TEXT NOT NULL,
+     signature TEXT NOT NULL,
+     created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
+   );
+   ```
+
+### Phase 2: RFC 6962 Cryptographic Primitives (`server/audit_merkle.ts`)
+
+1. Create `server/audit_merkle.ts`.
+2. Implement RFC 6962 leaf hashing: `SHA-256(0x00 || entry_bytes)`.
+3. Implement internal tree hashing:
+   `SHA-256(0x01 || left_child || right_child)`.
+4. Implement the logic to construct the Merkle Tree from an array of leaves and
+   generate an inclusion proof (`verifyInclusionProof`).
+5. Ensure all crypto utilizes `crypto.subtle.digest('SHA-256', ...)`.
+
+### Phase 3: The Micro-Batcher & STH Signing (`server/audit.ts`)
+
+1. Refactor `auditLog` to compute the `leaf_hash` synchronously before the async
+   database insert.
+2. Implement a background micro-batcher in `server/audit.ts` (or
+   `server/audit_merkle.ts` if better decoupled) using `setInterval` (e.g.,
+   every 1000ms, or on explicit flush).
+3. The batcher will:
+   - Query new `leaf_hash` entries from the database since the last STH
+     `tree_size`.
+   - Compute the new Merkle Root.
+   - Access the SPIFFE server signing key.
+   - Sign the STH (tree size, root hash) using `crypto.subtle.sign` (Ed25519).
+   - Insert the new STH into the `audit_sths` table.
+
+### Phase 4: Valkey STH Broadcast (`server/valkey.ts`)
+
+1. Once the STH is successfully saved to PostgreSQL, update the cache.
+2. Use `valkey.set('auth:audit:latest_sth', json_payload)`.
+3. Broadcast the update to independent witness nodes via pub/sub:
+   `valkey.publish('auth:audit:sth', json_payload)`.
+
+### Phase 5: Testing & Quality Gates
+
+1. Write hermetic tests in `server/audit_merkle.test.ts`.
+2. Use `@std/testing/mock` to mock `valkey`, `sqlWrapper`, and SPIFFE keys.
+3. Validate leaf hashing correctness against RFC 6962 test vectors.
+4. Verify the Valkey `PUBLISH` payload formatting.
+5. Execute standard quality gates: `deno fmt`, `deno task lint`,
+   `deno task check`, and `deno task test`.