auth-yes/tasks/complete/2026-0824.01.jul.feat.audit-merkle.add-merkle-tree-ledger-0547.md

# TASK METADATA

- **Target Files:** `server/audit.ts`, `server/audit_merkle.ts`,
  `server/valkey.ts`, `server/db.ts`, and database schema/migration files.
- **Core Objective:** Implement an append-only RFC 6962 Merkle Tree audit ledger
  in PostgreSQL with an in-process Deno micro-batcher for STH computation and
  Valkey pub/sub broadcast.
- **Dependencies:** Deno 2.x WebCrypto (Ed25519), Valkey connection, PostgreSQL
  schema updates, and SPIFFE workload identity key material.
- **Additional Important Notes:** Must use Deno's native WebCrypto (no external
  crypto npm packages). Valkey session keys should use raw strings without
  'session:' prefix (not fully applicable here, but good standard). Ensure tests
  mock Valkey, PostgreSQL, and SPIFFE cleanly.

---

## Architectural Considerations & Risks

Before implementing the RFC 6962 Merkle Tree Ledger, we must evaluate the
current architecture and explicitly identify risks, regressions, and necessary
constraints.

**Risks & Tradeoffs:**

1. **In-Process Micro-Batcher Memory & State:** The micro-batcher runs
   in-process inside Deno using asynchronous intervals (e.g., `setInterval` or
   explicit flush). If the Deno process crashes abruptly, unflushed audit logs
   in memory could be lost before the batch is committed to PostgreSQL and the
   Merkle tree head is updated.
   - _Mitigation:_ Ensure `auditLog` still performs an immediate async insert of
     the _record_ to the database (as it currently does via `server/audit.ts`),
     but the `leaf_hash` and tree updates are computed dynamically or buffered
     for the batched STH computation. The database remains the source of truth;
     the batcher just finalizes the cryptographical proof.
2. **Database Schema Locking:** Adding a `leaf_hash` column to the
   `audit_records` table and creating the `audit_sths` table might cause locking
   if the table is large.
   - _Mitigation:_ The `leaf_hash` can be computed prior to insert.
3. **SPIFFE Identity Dependency:** STH generation requires the Deno process to
   sign using the SPIFFE workload identity/server signing key via WebCrypto. If
   SPIFFE/FFI fails or the keys are rotating during a batch, the STH generation
   might fail.
   - _Mitigation:_ Graceful error handling and retry mechanism for STH signing.
4. **No Third-Party Dependencies:** Adhering to the memory context, we must
   exclusively use Deno 2.x native `crypto.subtle` API for SHA-256 and
   Ed25519/ECDSA, rather than any Node.js polyfills or npm packages.

**Alternatives Evaluated:**

- _External Worker:_ Using a dedicated worker (e.g., a separate service or CRON)
  for computing the Merkle root. _Decision:_ We rejected this to keep the
  application self-contained and zero-dependency, per user guidance.
- _Synchronous Tree Updates:_ Recomputing the tree on every single write.
  _Decision:_ Rejected. This would severely block relational write throughput. A
  micro-batcher (e.g., 30-60s or size threshold) is the optimal path for
  decoupling heavy crypto logic from the request hot-path.

## Proposed Implementation

The execution of this feature will be broken down into the following structured
phases:

### Phase 1: Database Schema Expansion

1. Update PostgreSQL schema to alter `audit_records` and add a `leaf_hash TEXT`
   column.
2. Create the new `audit_sths` table:
   ```sql
   CREATE TABLE audit_sths (
     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
     tree_size BIGINT NOT NULL,
     root_hash TEXT NOT NULL,
     signature TEXT NOT NULL,
     created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
   );
   ```

### Phase 2: RFC 6962 Cryptographic Primitives (`server/audit_merkle.ts`)

1. Create `server/audit_merkle.ts`.
2. Implement RFC 6962 leaf hashing: `SHA-256(0x00 || entry_bytes)`.
3. Implement internal tree hashing:
   `SHA-256(0x01 || left_child || right_child)`.
4. Implement the logic to construct the Merkle Tree from an array of leaves and
   generate an inclusion proof (`verifyInclusionProof`).
5. Ensure all crypto utilizes `crypto.subtle.digest('SHA-256', ...)`.

### Phase 3: The Micro-Batcher & STH Signing (`server/audit.ts`)

1. Refactor `auditLog` to compute the `leaf_hash` synchronously before the async
   database insert.
2. Implement a background micro-batcher in `server/audit.ts` (or
   `server/audit_merkle.ts` if better decoupled) using `setInterval` (e.g.,
   every 1000ms, or on explicit flush).
3. The batcher will:
   - Query new `leaf_hash` entries from the database since the last STH
     `tree_size`.
   - Compute the new Merkle Root.
   - Access the SPIFFE server signing key.
   - Sign the STH (tree size, root hash) using `crypto.subtle.sign` (Ed25519).
   - Insert the new STH into the `audit_sths` table.

### Phase 4: Valkey STH Broadcast (`server/valkey.ts`)

1. Once the STH is successfully saved to PostgreSQL, update the cache.
2. Use `valkey.set('auth:audit:latest_sth', json_payload)`.
3. Broadcast the update to independent witness nodes via pub/sub:
   `valkey.publish('auth:audit:sth', json_payload)`.

### Phase 5: Testing & Quality Gates

1. Write hermetic tests in `server/audit_merkle.test.ts`.
2. Use `@std/testing/mock` to mock `valkey`, `sqlWrapper`, and SPIFFE keys.
3. Validate leaf hashing correctness against RFC 6962 test vectors.
4. Verify the Valkey `PUBLISH` payload formatting.
5. Execute standard quality gates: `deno fmt`, `deno task lint`,
   `deno task check`, and `deno task test`.