docs: add comprehensive deep research and architecture brief for cross-ecosystem investigation

This commit is contained in:
Tyler Gillispie 2026-08-23 20:06:55 -07:00
parent e81969d5a4
commit 2684e30a23

436
docs/DEEP_RESEARCH_BRIEF.md Normal file
View File

@ -0,0 +1,436 @@
# Autonomous Zero-Trust IAM & Application Mesh: Deep Research & Architectural Master Brief
**Document Classification:** Strategic Architecture & Deep Research Dossier\
**Ecosystem Projects:** `auth-yes` (Zero-Trust IAM Authority) & `ed-droid`
(High-Velocity Telemetry Consumer)\
**Target Audience:** Principal Systems Architects, Cryptographic Engineers &
Deep AI Researchers\
**Version:** 3.0.0 (Master Research Edition)\
**Date:** August 2026
---
## Table of Contents
1. [Executive Summary & Architectural Paradigm](#1-executive-summary--architectural-paradigm)
2. [Complete Trajectory of Strategic Exchanges (Historical Archive)](#2-complete-trajectory-of-strategic-exchanges-historical-archive)
- [Exchange 1: Initial Consumer Audit, Gap Analysis & Upstream RFCs](#exchange-1-initial-consumer-audit-gap-analysis--upstream-rfcs)
- [Exchange 2: Core Authority Verdict, Zero-Trust Defense & Upstream SDK Upgrades](#exchange-2-core-authority-verdict-zero-trust-defense--upstream-sdk-upgrades)
- [Exchange 3: The 6 Deep Discovery Questions on Identity Boundaries](#exchange-3-the-6-deep-discovery-questions-on-identity-boundaries)
- [Exchange 4: The 2 Boundary-Pushing Paradigms (Ingress Hydration & Passkey Mesh)](#exchange-4-the-2-boundary-pushing-paradigms-ingress-hydration--passkey-mesh)
- [Exchange 5: Strategic Resolutions (Flattened Vectors, Token Tiering & Ghost Cockpit UX)](#exchange-5-strategic-resolutions-flattened-vectors-token-tiering--ghost-cockpit-ux)
3. [System Capabilities & Architecture Accomplishments](#3-system-capabilities--architecture-accomplishments)
4. [Competitive Landscape & Industry State-of-the-Art Analysis](#4-competitive-landscape--industry-state-of-the-art-analysis)
5. [Deep Research Frontiers & Collaborative Directives](#5-deep-research-frontiers--collaborative-directives)
---
## 1. Executive Summary & Architectural Paradigm
The **Auth-Yes / ed-droid** ecosystem represents a generational leap in Identity
and Access Management (IAM) and zero-trust microservice architecture. By
rejecting the protocol bloat, redirect latency, and security compromises of
traditional OIDC/OAuth2/SAML identity providers (e.g., Keycloak, Authentik,
Okta), Auth-Yes delivers:
1. **Pure WebAuthn Passkey Authority:** 100% phishing-proof asymmetric
authentication scoped to the parent domain (`RP_ID=atyg.org`), sharing
credentials seamlessly across all subdomains (`*.atyg.org`).
2. **The "Two-Locks" Defense-in-Depth Model:**
- **Lock 1 (Perimeter Guard / Traefik ForwardAuth):** Universal edge-level
session inspection, blocking unauthenticated bot traffic and scanners in
microseconds and automating UI redirects.
- **Lock 2 (Zero-Trust Core / In-App SDK):** In-app memory token validation
($<30\mu s$) backed by **Valkey 8 RESP3 Client-Side Cache Tracking
(BCAST)** with sub-millisecond real-time push invalidation.
3. **Decoupled Workload Identity (SPIFFE/SPIRE):** Cryptographic container
identity attestation over UNIX domain sockets (`spire-socket`), securing
internal ConnectRPC / HTTP/2 communications with zero-trust mTLS.
4. **Decoupled Default-Deny RBAC:** Central identity is completely decoupled
from application-specific databases. A user possesses zero application
permissions until an explicit row is granted in the central `grants` table.
```
[ Public Internet / Pilot Browser ]
1. WebAuthn Passkey Login
2. Wildcard Cookie: *.atyg.org
═════════════════════════════════════════
Ingress Reverse Proxy (Traefik Ingress)
═════════════════════════════════════════
┌────────────────────┴────────────────────┐
▼ Router 1: Priority 10 (Secure Web) ▼ Router 2: Priority 100 (Public Edge)
[ ForwardAuth Interception ] [ Public Bypass / Direct ]
Query http://auth-api:8000/api/forward-auth (CLI scripts, Webhooks, /health)
- 200 OK -> Injects X-Forwarded-User-* │
- 401 Unauthorized -> Redirects to /login │
│ │
└────────────────────┬────────────────────┘
═════════════════════════════════════════════════════════════════════
Downstream Application Container (ed-droid-hub: Port 8000)
├── Byte-1 SSR UI Hydration (reads Ingress Grant Headers directly)
├── In-App Zero-Trust Session Middleware (@auth-yes/sdk)
│ ├── In-Memory L1 Cache (<30µs evaluation)
│ └── ConnectRPC Fallback (http://auth-api:8000)
└── Telemetry Stream & Non-Destructive "Ghost Cockpit" WebSockets
═════════════════════════════════════════════════════════════════════
Auth-Yes Central Mesh (`auth-internal-net`)
├── auth-api:8000 -> ConnectRPC Gateway & PostgreSQL 18 RBAC Engine
├── auth-valkey:6379 -> RESP3 Broadcast Client-Side Invalidation Bus
└── spire-agent.sock -> SPIFFE Workload API Cryptographic Attestation
═════════════════════════════════════════════════════════════════════
```
---
## 2. Complete Trajectory of Strategic Exchanges (Historical Archive)
The following historical record documents the exact collaborative evolution
between the **ed-droid Solutions Architecture Team** and the **Auth-Yes Core
Systems Team**:
```
=== DIALOGUE ARCHIVE START ===
```
### Exchange 1: Initial Consumer Audit, Gap Analysis & Upstream RFCs
_(From: ed-droid Architecture Team | To: Auth-Yes Core Team)_
> **Context:** Following the standalone extraction of `auth-yes`, `ed-droid`
> conducted a full gap analysis against `ONBOARDING.md` and submitted three
> upstream RFCs:
>
> 1. **RFC 1 (Direct Scope Attachment in `@auth-yes/sdk/hono`):**\
> `sdk/hono.ts` only sets `userId`. Consuming applications are forced to
> write custom middleware to access `scopes`. Proposed setting
> `c.set("userId", session.uuid)` and
> `c.set("scopes", session.scopes || [])`.
> 2. **RFC 2 (ForwardAuth Ingress Header Fast-Path):**\
> Proposed adding `trustForwardHeader?: boolean` to allow microservices to
> trust `X-Forwarded-User-Id` directly from Traefik without re-validating the
> token in-app.
> 3. **RFC 3 (ConnectRPC Transport Timeout Configuration):**\
> Proposed adding `timeoutMs?: number` with `AbortSignal.timeout(timeoutMs)`
> to prevent client requests from hanging during gateway restarts.
> 4. **Traefik Dual-Router Pattern:**\
> Identified that global ForwardAuth blocks headless Fedora gaming PC daemons
> from downloading CLI scripts (`/edge/daemon.ts`,
> `/install-edge-service.ts`). Proposed splitting Traefik into a Priority 100
> Public Bypass Router and a Priority 10 Authenticated Secure Router.
---
### Exchange 2: Core Authority Verdict, Zero-Trust Defense & Upstream SDK Upgrades
_(From: Auth-Yes Core Team | To: ed-droid Architecture Team)_
> **Core Authority Assessment:**
>
> 1. **RFC 1 (Approved — 10/10):** Implemented immediately. `sdk/hono.ts` now
> automatically injects `userId` and `scopes`, and exports a ready-to-use
> `requireScope(role)` middleware guard.
> 2. **RFC 2 (Rejected / Caution — Zero-Trust Risk):**\
> **Verdict:** In a Zero-Trust architecture, microservices must **never
> blindly trust unauthenticated HTTP headers**. If an attacker or lateral
> container on the bridge network sends an internal request with
> `X-Forwarded-User-Id: <victim-uuid>`, the app would be compromised. Because
> Valkey RESP3 L1 memory cache checks take **$< 30 \mu s$**, in-app token
> verification is essentially free and mandatory.
> 3. **RFC 3 (Approved — 10/10):** Implemented immediately. Added `timeoutMs`
> (default: 5000ms) with `AbortSignal.timeout` to `@auth-yes/sdk`.
> 4. **Traefik Dual-Router Standardization:** Formally codified the Priority 100
> Public / Priority 10 Secure router standard into `COMPOSE_CONVENTIONS.md`.
---
### Exchange 3: The 6 Deep Discovery Questions on Identity Boundaries
_(From: ed-droid Architecture Team | To: Auth-Yes Core Team)_
> **Discovery Questions Submitted by ed-droid:**
>
> 1. _Cross-Checking Headers vs. Token Payloads:_ Does `@auth-yes/sdk` strictly
> ignore the header in favor of the token, or should consuming apps
> cross-check them (e.g. logging a security alert on mismatch)?
> 2. _Isolating Tier 2 (Off-the-Shelf) Apps:_ For third-party off-the-shelf apps
> (like Grafana or Portainer) that cannot run `@auth-yes/sdk` and must rely
> purely on `X-Forwarded-User`, what is the recommended network isolation
> pattern to prevent another container on the bridge network from spoofing
> that header directly to the upstream app?
> 3. _Ephemeral Signed Headers for Non-Redis Environments:_ For lightweight
> microservices that cannot connect to Valkey/Redis for L1 tracking, has
> Auth-Yes considered an optional signed header standard (e.g. Traefik
> ForwardAuth injecting an HMAC or Ed25519-signed `X-Forwarded-Sig` verified
> via public key in <5μs)?
> 4. _WebSockets & Real-Time Revocation (Live Telemetry):_ `ed-droid` maintains
> persistent WebSocket connections for live Elite Dangerous telemetry.
> Traefik ForwardAuth validates the handshake at connect time. When a session
> is revoked in Valkey, what is Auth-Yes's recommended pattern for
> microservices to proactively terminate active WebSocket streams?
> 5. _Workload Identity (mTLS & SPIFFE) over ConnectRPC:_ For internal
> microservices calling `auth-api:8000` via ConnectRPC over HTTP/2, how will
> Auth-Yes attest caller identity now that `spire_ffi` is encapsulated? Is
> the recommended path passing client TLS certificates via Node/Deno
> `tlsCert`/`tlsKey` options in `createConnectTransport`?
> 6. _The "Two-Locks" Defense-in-Depth Standard:_ Does Auth-Yes endorse the
> "Two-Locks" architecture (Lock 1: Traefik ForwardAuth Edge Ingress + Lock
> 2: In-App SDK L1 Token Validation) as the official gold-standard reference
> pattern in `ONBOARDING.md`?
---
### Exchange 4: The 2 Boundary-Pushing Paradigms (Ingress Hydration & Passkey Mesh)
_(From: Auth-Yes Core Team | To: ed-droid Architecture Team)_
> **Auth-Yes Architectural Formulations:**
>
> #### Paradigm 4: Ingress Grant Injection for Zero-Latency SSR Hydration
>
> Auth-Yes eliminates the secondary `/api/me` or `/api/permissions` client
> fetch. When Traefik queries ForwardAuth, Auth-Yes maps `X-Forwarded-Host` to
> `app_id` and Valkey user grants, injecting `X-Forwarded-User-Id`,
> `X-Forwarded-User-Name`, and `X-Forwarded-Scopes`. The downstream SSR engine
> renders 100% role-tailored HTML on Byte 1 with zero layout shift!
>
> #### Paradigm 5: Autonomous Multi-Device Passkey Mesh & Zero-Downgrade Recovery
>
> Auth-Yes solves the fatal flaw of the passkey industry (SMS/email fallback
> downgrade). It establishes:
>
> 1. **Multi-Passkey Mesh ($N \ge 2$):** Primary YubiKey + iPhone/MacBook Touch
> ID; any active key can enroll new authenticators.
> 2. **Armored Offline Cryptographic Vouchers:** 24-word BIP-39 seed / Shamir
> split stored in cold storage; verified via zero-knowledge challenge without
> server-stored private secrets.
> 3. **Multi-Party Out-of-Band Admin Attestation:** Admin issues 24-hour
> single-use cryptographic recovery link delivered via secure out-of-band
> channels.
>
> #### 3 Strategic Questions Posed Back to ed-droid:
>
> 1. _Grant Vector Header Schema:_ Flattened headers
> (`X-Forwarded-Scopes: commander,operator`) vs single base64url JSON claims?
> 2. _Multi-Tenant Edge Node Voucher Lifecycle:_ Central Auth-Yes device
> management vs ed-droid operational authority?
> 3. _Client-Side Live Revocation UX:_ Handling WebSocket termination without
> destroying in-flight telemetry state?
---
### Exchange 5: Strategic Resolutions (Flattened Vectors, Token Tiering & Ghost Cockpit UX)
_(From: ed-droid Architecture Team | To: Auth-Yes Core Team)_
> **ed-droid Strategic Resolutions:**
>
> 1. **Grant Vector Header Schema $\rightarrow$ Standard Flattened Headers:**
> - _Native Traefik Ingress:_ Traefik `authResponseHeaders` captures
> individual headers with zero plugin overhead.
> - _Zero-Overhead SSR:_ `c.req.header("X-Forwarded-Scopes")?.split(",")`
> requires zero JSON/base64 CPU cycles.
> - _Observability:_ Readable directly in access logs and distributed tracing
> spans.
> 2. **Multi-Tenant Edge Node Voucher Lifecycle $\rightarrow$ Tiered Domain
> Authority:**
> - _Auth-Yes:_ Authenticates the pilot and asserts the `commander`/`admin`
> role.
> - _ed-droid:_ Issues, hashes (SHA-256), and manages the operational
> telemetry streaming socket.
> - _Unified Revocation:_ Disabling the pilot in Auth-Yes cascades instantly
> to disconnect all edge nodes bound to that `user_id`.
> 3. **Client-Side Live Revocation UX $\rightarrow$ The "Ghost Cockpit"
> Protocol:**
> - _Problem with Anti-Pattern:_ Full browser redirect to `/login` destroys
> unsaved flight routes, active market data, and exploration scans.
> - _The Ghost Cockpit Protocol:_
> 1. Server intercepts Valkey session purge and emits WebSocket frame
> `{ "type": "AUTH_REVOKED" }` before closing TCP connection.
> 2. Frontend executes **Non-Destructive State Freeze** (freezes cockpit
> telemetry in memory without blanking screen).
> 3. Displays an ambient glassmorphism dialog: _"Flight Link Suspended. Tap
> Passkey / Touch ID to Re-engage."_
> 4. Pilot taps passkey $\rightarrow$ background WebAuthn ceremony executes
> $\rightarrow$ new cookie set $\rightarrow$ WebSocket reconnects
> seamlessly with **zero lost data**!
```
=== DIALOGUE ARCHIVE END ===
```
---
## 3. System Capabilities & Architecture Accomplishments
The combined implementation across `auth-yes` and `ed-droid` establishes several
foundational capabilities:
| Capability Domain | Implementation Mechanism | Latency / Security Benchmark | Accomplished State |
| :---------------------------- | :----------------------------------------------- | :----------------------------------------- | :----------------- |
| **Passkey Authority** | SimpleWebAuthn + Deno 2 Web Cryptography | Zero passwords, zero phishing vectors | Fully Operational |
| **Domain Cookie Scoping** | Wildcard `.atyg.org` with Lax / Secure flags | Shared across all subdomains automatically | Fully Operational |
| **Edge Ingress Protection** | Traefik ForwardAuth (Tier 1 & 2) | Microsecond edge rejection; 0 CPU on app | Fully Operational |
| **In-App Zero-Trust (L1)** | Valkey 8 RESP3 `CLIENT TRACKING BCAST` | **$< 30$ microseconds** in-memory lookup | Fully Operational |
| **Instant Global Revocation** | Valkey RESP3 Push Invalidation Bus | **$< 1$ millisecond** cluster-wide purge | Fully Operational |
| **Workload Attestation** | SPIFFE/SPIRE 1.9 + Rust FFI Crate (`spire_ffi`) | Cryptographic X.509 SVID container auth | Fully Operational |
| **Byte-1 SSR Hydration** | ForwardAuth Grant Vector Header Injection | Sub-millisecond TTFB; zero layout shift | Design Approved |
| **Non-Destructive UX** | "Ghost Cockpit" WebSocket Freeze + Modal Re-Auth | Zero state loss on session rotation/expiry | Design Approved |
| **Smart Image Lifecycle** | Alpine wrapper + auto-join token bootstrapping | Self-seeding configs; zero compose scripts | Fully Operational |
---
## 4. Competitive Landscape & Industry State-of-the-Art Analysis
To understand why this architecture is generational, we compare Auth-Yes against
the industry's leading identity platforms:
```
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ IAM & ZERO-TRUST EVOLUTION MATRIX │
├──────────────────────┬──────────────────────┬──────────────────────┬──────────────────────┬─────────────────┤
│ Architecture Metric │ Legacy Monoliths │ Modern Headless IAM │ Cloud Edge ZeroTrust │ Auth-Yes Fabric │
│ │ (Keycloak / Okta) │ (Ory Kratos/Zitadel) │ (Cloudflare / Pomer) │ (Current System)│
├──────────────────────┼──────────────────────┼──────────────────────┼──────────────────────┼─────────────────┤
│ Protocol Weight │ OIDC / SAML 2.0 │ JSON REST / OIDC │ Proprietary Proxy │ ConnectRPC/HTTP2│
│ Browser Friction │ 3-5 Redirect Loops │ REST Form Submits │ Tunnel Interception │ Zero-Redirect │
│ Client Invalidation │ Polling / JWT Expiry │ Database Query Loop │ Short-lived Cookies │ RESP3 Push Bus │
│ In-Memory L1 Cache │ None (Remote DB/JWT) │ Optional Redis Read │ Edge Edge-Cache only │ <30µs RESP3 L1
│ Workload Attestation │ Static Client Secret │ Service Account JWT │ WireGuard / mTLS │ SPIFFE/SPIRE FFI│
│ Recovery Fallback │ Insecure SMS / Email │ Insecure SMS / Email │ Email Magic Link │ Zero-Downgrade │
│ SSR Hydration Cost │ Secondary /api/me │ Secondary Fetch │ Header Injection │ Ingress Vector │
│ Container Footprint │ 1.5GB+ JVM / Heavy │ 200MB Go Monolith │ Proprietary Daemon │ Minimal Alpine │
└──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴─────────────────┘
```
### Deep Analysis of Competitive Approaches:
1. **Keycloak / Okta / Authentik (Legacy OIDC Monoliths):**
- _The Problem:_ Heavyweight redirect dances (`/authorize` $\rightarrow$
`/callback` $\rightarrow$ `/token`). Every downstream app must exchange
authorization codes for access tokens, verify RSA signatures, and deal with
un-revokable JWT expiration windows.
- _How Auth-Yes Wins:_ Direct parent-domain session scoping + ConnectRPC
removes all redirect latency. Session validation takes $<30\mu s$ in memory
instead of 50ms of network hops.
2. **Ory Kratos / Zitadel (Headless Open-Source IAM):**
- _The Problem:_ Still rely on traditional database query loops or standard
Redis polling for session validation. They lack native SPIFFE/SPIRE
workload attestation for microservice-to-microservice mesh security.
- _How Auth-Yes Wins:_ Combines WebAuthn passkeys, Valkey RESP3 push
invalidation, and hardware SPIFFE container attestation into a unified,
zero-dependency Deno SDK.
3. **Cloudflare Access / Tailscale / Pomerium (Edge Proxy Identity):**
- _The Problem:_ Provide perimeter protection (Lock 1) but lack
application-level RBAC integration (Lock 2). They treat identity as a
binary gate at the edge, forcing downstream apps to maintain separate user
permission databases.
- _How Auth-Yes Wins:_ The "Two-Locks" model provides edge perimeter security
_and_ deep application RBAC scope vectors injected directly into
application contexts.
---
## 5. Deep Research Frontiers & Collaborative Directives
To push this ecosystem to the absolute technological limit, we have structured
**10 core research frontiers** for exploratory investigation:
### Frontier 1: Zero-Knowledge Multi-Authenticator Mesh & Cryptographic Quorums
- **Research Objective:** Formalize the mathematical protocol for multi-device
WebAuthn enrollment where any $M$-of-$N$ active authenticators can
cryptographically authorize new credentials without an intermediate server
having access to private keys.
- **Key Challenge:** Designing an armored 24-word BIP-39 recovery ceremony that
executes zero-knowledge challenge responses entirely within the client
WebAssembly/WebCrypto runtime.
### Frontier 2: Ephemeral Ed25519-Signed ForwardAuth Ingress Headers
- **Research Objective:** For lightweight edge microservices running outside the
internal Valkey mesh (e.g. serverless functions, remote IoT daemons),
establish an asymmetric signed header standard:
```http
X-Auth-User-Id: <uuid>
X-Auth-Scopes: <scopes>
X-Auth-Sig: t=<timestamp>;sig=<ed25519_signature>
```
- **Key Challenge:** Ensuring sub-5µs public key verification with strict
timestamp drift bounds ($\pm 30$s) and anti-replay nonce tracking.
### Frontier 3: Zero-Copy SPIRE SVID Rotation & High-Concurrency FFI Ingestion
- **Research Objective:** Investigate zero-copy memory mapping between the Rust
`spire_ffi` crate and Deno runtime so that periodic SVID rotations (e.g.,
1-hour certificates) update in-flight HTTP/2 mTLS client connections with zero
TCP reconnection latency or memory fragmentation.
### Frontier 4: The "Ghost Cockpit" Standard: Seamless Re-Auth within Live WebSockets / WebTransport
- **Research Objective:** Standardize the non-destructive WebSocket/WebTransport
session renewal protocol across diverse UI ecosystems (React, Svelte, Vue,
Solid).
- **Key Challenge:** Defining the precise control frame choreography so that
background WebAuthn passkey attestation transparently upgrades active
bidirectional streaming connections without buffering overflows or data loss.
### Frontier 5: Parent-Domain Scoping (`RP_ID`) vs. Cross-TLD Related Origin Requests (ROR)
- **Research Objective:** Evaluate W3C Related Origin Requests
(`https://auth.<domain>/.well-known/webauthn`) for multi-domain federation
(e.g. bridging `atyg.org` with third-party domain extensions) while
maintaining single-passkey enrollment.
### Frontier 6: High-Throughput Sliding-Window Rate Limiting over Valkey Clusters
- **Research Objective:** Benchmark distributed sliding-window log algorithms in
Valkey 8 under simulated DDoS conditions ($100,000+$ req/sec) to ensure public
WebAuthn challenge endpoints remain resilient against distributed resource
exhaustion attacks.
### Frontier 7: Immutable Cryptographic Audit Ledger with Merkle Tree Attestation
- **Research Objective:** Upgrade the internal `audit_records` table into a
verifiable tamper-evident hash chain (Merkle Tree), where hourly root hashes
are cryptographically signed by SPIRE server keys, guaranteeing mathematical
non-repudiation for security compliance.
### Frontier 8: Sub-10µs Dynamic Multi-Tenant RBAC Matrix Resolution
- **Research Objective:** Optimize the PostgreSQL `grants` cache indexing and
Valkey pipelined hash storage so that user permissions across hundreds of
registered microservices evaluate in $<10\mu s$ per RPC call.
### Frontier 9: Container Ingress Boundary Hardening & Lateral Network Isolation
- **Research Objective:** Formalize Docker and Podman network topology
constraints to guarantee that Tier 2 off-the-shelf containers (Grafana,
Portainer) have zero network paths to internal microservice meshes, preventing
lateral header spoofing.
### Frontier 10: Universal Language SDK Implementations (Go, Rust, Python, Deno)
- **Research Objective:** Define the standardized multi-language architecture
for `@auth-yes/sdk` across Go, Rust, and Python, ensuring consistent
ConnectRPC transport, timeout handling, and RESP3 client-side cache tracking
across all microservice runtimes.
---
## 6. Document Metadata & Distribution
- **Master Repository:** `auth-yes` (`git@github.com:mrteye/auth-yes.git` /
`git@git.atyg.org:tylerg/auth-yes.git`)
- **Consumer Repository:** `ed-droid` (`git@github.com:mrteye/ed-droid.git` /
`git@git.atyg.org:tylerg/ed-droid.git`)
- **Status:** Living Master Research Brief
- **Maintained By:** Auth-Yes IAM Core Architecture Team & ed-droid Solutions
Engineering