diff --git a/tasks/new/2026-0824.01.jul.feat.audit-merkle.add-merkle-tree-ledger-0547.md b/tasks/new/2026-0824.01.jul.feat.audit-merkle.add-merkle-tree-ledger-0547.md new file mode 100644 index 0000000..c2253d2 --- /dev/null +++ b/tasks/new/2026-0824.01.jul.feat.audit-merkle.add-merkle-tree-ledger-0547.md @@ -0,0 +1,117 @@ +# TASK METADATA + +- **Target Files:** `server/audit.ts`, `server/audit_merkle.ts`, + `server/valkey.ts`, `server/db.ts`, and database schema/migration files. +- **Core Objective:** Implement an append-only RFC 6962 Merkle Tree audit ledger + in PostgreSQL with an in-process Deno micro-batcher for STH computation and + Valkey pub/sub broadcast. +- **Dependencies:** Deno 2.x WebCrypto (Ed25519), Valkey connection, PostgreSQL + schema updates, and SPIFFE workload identity key material. +- **Additional Important Notes:** Must use Deno's native WebCrypto (no external + crypto npm packages). Valkey session keys should use raw strings without + 'session:' prefix (not fully applicable here, but good standard). Ensure tests + mock Valkey, PostgreSQL, and SPIFFE cleanly. + +--- + +## Architectural Considerations & Risks + +Before implementing the RFC 6962 Merkle Tree Ledger, we must evaluate the +current architecture and explicitly identify risks, regressions, and necessary +constraints. + +**Risks & Tradeoffs:** + +1. **In-Process Micro-Batcher Memory & State:** The micro-batcher runs + in-process inside Deno using asynchronous intervals (e.g., `setInterval` or + explicit flush). If the Deno process crashes abruptly, unflushed audit logs + in memory could be lost before the batch is committed to PostgreSQL and the + Merkle tree head is updated. + - _Mitigation:_ Ensure `auditLog` still performs an immediate async insert of + the _record_ to the database (as it currently does via `server/audit.ts`), + but the `leaf_hash` and tree updates are computed dynamically or buffered + for the batched STH computation. The database remains the source of truth; + the batcher just finalizes the cryptographical proof. +2. **Database Schema Locking:** Adding a `leaf_hash` column to the + `audit_records` table and creating the `audit_sths` table might cause locking + if the table is large. + - _Mitigation:_ The `leaf_hash` can be computed prior to insert. +3. **SPIFFE Identity Dependency:** STH generation requires the Deno process to + sign using the SPIFFE workload identity/server signing key via WebCrypto. If + SPIFFE/FFI fails or the keys are rotating during a batch, the STH generation + might fail. + - _Mitigation:_ Graceful error handling and retry mechanism for STH signing. +4. **No Third-Party Dependencies:** Adhering to the memory context, we must + exclusively use Deno 2.x native `crypto.subtle` API for SHA-256 and + Ed25519/ECDSA, rather than any Node.js polyfills or npm packages. + +**Alternatives Evaluated:** + +- _External Worker:_ Using a dedicated worker (e.g., a separate service or CRON) + for computing the Merkle root. _Decision:_ We rejected this to keep the + application self-contained and zero-dependency, per user guidance. +- _Synchronous Tree Updates:_ Recomputing the tree on every single write. + _Decision:_ Rejected. This would severely block relational write throughput. A + micro-batcher (e.g., 30-60s or size threshold) is the optimal path for + decoupling heavy crypto logic from the request hot-path. + +## Proposed Implementation + +The execution of this feature will be broken down into the following structured +phases: + +### Phase 1: Database Schema Expansion + +1. Update PostgreSQL schema to alter `audit_records` and add a `leaf_hash TEXT` + column. +2. Create the new `audit_sths` table: + ```sql + CREATE TABLE audit_sths ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + tree_size BIGINT NOT NULL, + root_hash TEXT NOT NULL, + signature TEXT NOT NULL, + created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW() + ); + ``` + +### Phase 2: RFC 6962 Cryptographic Primitives (`server/audit_merkle.ts`) + +1. Create `server/audit_merkle.ts`. +2. Implement RFC 6962 leaf hashing: `SHA-256(0x00 || entry_bytes)`. +3. Implement internal tree hashing: + `SHA-256(0x01 || left_child || right_child)`. +4. Implement the logic to construct the Merkle Tree from an array of leaves and + generate an inclusion proof (`verifyInclusionProof`). +5. Ensure all crypto utilizes `crypto.subtle.digest('SHA-256', ...)`. + +### Phase 3: The Micro-Batcher & STH Signing (`server/audit.ts`) + +1. Refactor `auditLog` to compute the `leaf_hash` synchronously before the async + database insert. +2. Implement a background micro-batcher in `server/audit.ts` (or + `server/audit_merkle.ts` if better decoupled) using `setInterval` (e.g., + every 1000ms, or on explicit flush). +3. The batcher will: + - Query new `leaf_hash` entries from the database since the last STH + `tree_size`. + - Compute the new Merkle Root. + - Access the SPIFFE server signing key. + - Sign the STH (tree size, root hash) using `crypto.subtle.sign` (Ed25519). + - Insert the new STH into the `audit_sths` table. + +### Phase 4: Valkey STH Broadcast (`server/valkey.ts`) + +1. Once the STH is successfully saved to PostgreSQL, update the cache. +2. Use `valkey.set('auth:audit:latest_sth', json_payload)`. +3. Broadcast the update to independent witness nodes via pub/sub: + `valkey.publish('auth:audit:sth', json_payload)`. + +### Phase 5: Testing & Quality Gates + +1. Write hermetic tests in `server/audit_merkle.test.ts`. +2. Use `@std/testing/mock` to mock `valkey`, `sqlWrapper`, and SPIFFE keys. +3. Validate leaf hashing correctness against RFC 6962 test vectors. +4. Verify the Valkey `PUBLISH` payload formatting. +5. Execute standard quality gates: `deno fmt`, `deno task lint`, + `deno task check`, and `deno task test`.