auth-yes/tasks/complete/2026-0824.01.jul.feat.audit-merkle.add-merkle-tree-ledger-0547.md

TASK METADATA

• Target Files: server/audit.ts, server/audit_merkle.ts, server/valkey.ts, server/db.ts, and database schema/migration files.
• Core Objective: Implement an append-only RFC 6962 Merkle Tree audit ledger in PostgreSQL with an in-process Deno micro-batcher for STH computation and Valkey pub/sub broadcast.
• Dependencies: Deno 2.x WebCrypto (Ed25519), Valkey connection, PostgreSQL schema updates, and SPIFFE workload identity key material.
• Additional Important Notes: Must use Deno's native WebCrypto (no external crypto npm packages). Valkey session keys should use raw strings without 'session:' prefix (not fully applicable here, but good standard). Ensure tests mock Valkey, PostgreSQL, and SPIFFE cleanly.

---

Architectural Considerations & Risks

Before implementing the RFC 6962 Merkle Tree Ledger, we must evaluate the current architecture and explicitly identify risks, regressions, and necessary constraints.

Risks & Tradeoffs:

1. In-Process Micro-Batcher Memory & State: The micro-batcher runs in-process inside Deno using asynchronous intervals (e.g., setInterval or explicit flush). If the Deno process crashes abruptly, unflushed audit logs in memory could be lost before the batch is committed to PostgreSQL and the Merkle tree head is updated.
   • Mitigation: Ensure auditLog still performs an immediate async insert of the record to the database (as it currently does via server/audit.ts), but the leaf_hash and tree updates are computed dynamically or buffered for the batched STH computation. The database remains the source of truth; the batcher just finalizes the cryptographical proof.
2. Database Schema Locking: Adding a leaf_hash column to the audit_records table and creating the audit_sths table might cause locking if the table is large.
   • Mitigation: The leaf_hash can be computed prior to insert.
3. SPIFFE Identity Dependency: STH generation requires the Deno process to sign using the SPIFFE workload identity/server signing key via WebCrypto. If SPIFFE/FFI fails or the keys are rotating during a batch, the STH generation might fail.
   • Mitigation: Graceful error handling and retry mechanism for STH signing.
4. No Third-Party Dependencies: Adhering to the memory context, we must exclusively use Deno 2.x native crypto.subtle API for SHA-256 and Ed25519/ECDSA, rather than any Node.js polyfills or npm packages.

Alternatives Evaluated:

• External Worker: Using a dedicated worker (e.g., a separate service or CRON) for computing the Merkle root. Decision: We rejected this to keep the application self-contained and zero-dependency, per user guidance.
• Synchronous Tree Updates: Recomputing the tree on every single write. Decision: Rejected. This would severely block relational write throughput. A micro-batcher (e.g., 30-60s or size threshold) is the optimal path for decoupling heavy crypto logic from the request hot-path.

Proposed Implementation

The execution of this feature will be broken down into the following structured phases:

Phase 1: Database Schema Expansion

1. Update PostgreSQL schema to alter audit_records and add a leaf_hash TEXT column.
2. Create the new audit_sths table:

   CREATE TABLE audit_sths (
     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
     tree_size BIGINT NOT NULL,
     root_hash TEXT NOT NULL,
     signature TEXT NOT NULL,
     created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
   );
   

Phase 2: RFC 6962 Cryptographic Primitives (server/audit_merkle.ts)

1. Create server/audit_merkle.ts.
2. Implement RFC 6962 leaf hashing: SHA-256(0x00 || entry_bytes).
3. Implement internal tree hashing: SHA-256(0x01 || left_child || right_child).
4. Implement the logic to construct the Merkle Tree from an array of leaves and generate an inclusion proof (verifyInclusionProof).
5. Ensure all crypto utilizes crypto.subtle.digest('SHA-256', ...).

Phase 3: The Micro-Batcher & STH Signing (server/audit.ts)

1. Refactor auditLog to compute the leaf_hash synchronously before the async database insert.
2. Implement a background micro-batcher in server/audit.ts (or server/audit_merkle.ts if better decoupled) using setInterval (e.g., every 1000ms, or on explicit flush).
3. The batcher will:
   • Query new leaf_hash entries from the database since the last STH tree_size.
   • Compute the new Merkle Root.
   • Access the SPIFFE server signing key.
   • Sign the STH (tree size, root hash) using crypto.subtle.sign (Ed25519).
   • Insert the new STH into the audit_sths table.

Phase 4: Valkey STH Broadcast (server/valkey.ts)

1. Once the STH is successfully saved to PostgreSQL, update the cache.
2. Use valkey.set('auth:audit:latest_sth', json_payload).
3. Broadcast the update to independent witness nodes via pub/sub: valkey.publish('auth:audit:sth', json_payload).

Phase 5: Testing & Quality Gates

1. Write hermetic tests in server/audit_merkle.test.ts.
2. Use @std/testing/mock to mock valkey, sqlWrapper, and SPIFFE keys.
3. Validate leaf hashing correctness against RFC 6962 test vectors.
4. Verify the Valkey PUBLISH payload formatting.
5. Execute standard quality gates: deno fmt, deno task lint, deno task check, and deno task test.