196 lines
9.8 KiB
Markdown
196 lines
9.8 KiB
Markdown
# Cloudflare Infrastructure Diagnostics & Telemetry Log
|
||
|
||
**Document Version:** 1.0.0
|
||
**Date:** 2026-08-22
|
||
**Target Domain:** `atyg.org` (Zone ID: `b8c97017e02a330f48460146d75cb8ad`)
|
||
**Account:** `Mrteye@gmail.com's Account` (Account ID: `9424f5616b98fb3c3ddabcb92e284c83`)
|
||
**Active Tunnel:** `carrot` (Tunnel ID: `1070fbdd-2dc0-45ea-8f5d-d11a8035c669`)
|
||
**Origin IP:** `98.97.38.125`
|
||
|
||
---
|
||
|
||
## 1. Executive Summary & Problem Statement
|
||
|
||
Users experienced intermittent connection timeouts, slow response times (12s+ hangs), and `HTTP 502 Bad Gateway` errors across services hosted on `atyg.org` (including `git.atyg.org`, `traefik.atyg.org`, and `drive-app.atyg.org`).
|
||
|
||
Comprehensive diagnostic probes and direct Cloudflare GraphQL / REST API queries confirmed that the issue was **not** caused by high traffic, Bot Management, or WAF rules, but rather by **two conflicting `cloudflared` client instances running concurrently on the origin host** and attached to the same tunnel ID (`carrot`).
|
||
|
||
---
|
||
|
||
## 2. Empirical Test Results & Telemetry
|
||
|
||
### 2.1 Multi-Probe Stress Test (Raw Evidence)
|
||
A 15-probe automated test against `https://git.atyg.org/tylerg/custom-nas.git/info/refs?service=git-upload-pack` revealed a 40–60% drop rate:
|
||
|
||
| Probe | Status | Response Time | Cloudflare Edge Ray | Outcome |
|
||
| :--- | :---: | :---: | :--- | :--- |
|
||
| #01 | 200 OK | 0.093s (93 ms) | `a2f621115b52c3ca-SEA` | Passed (Fast) |
|
||
| #02 | FAILED | 12.039s | N/A | Timed Out (Halted) |
|
||
| #03 | FAILED | 12.033s | N/A | Timed Out (Halted) |
|
||
| #04 | FAILED | 12.045s | N/A | Timed Out (Halted) |
|
||
| #05 | 200 OK | 0.101s (101 ms) | `a2f622004b667bf2-SEA` | Passed (Fast) |
|
||
| #06 | 200 OK | 0.101s (101 ms) | `a2f622040dc4a49b-SEA` | Passed (Fast) |
|
||
| #07 | 200 OK | 0.097s (97 ms) | `a2f62207ca4778c9-SEA` | Passed (Fast) |
|
||
| #08 | FAILED | 12.041s | N/A | Timed Out (Halted) |
|
||
| #09 | 200 OK | 0.101s (101 ms) | `a2f62259e8da7bf2-SEA` | Passed (Fast) |
|
||
| #10 | 200 OK | 0.098s (98 ms) | `a2f6225da8815b4d-SEA` | Passed (Fast) |
|
||
| #11 | 200 OK | 0.119s (119 ms) | `a2f622617db3764b-SEA` | Passed (Fast) |
|
||
| #12 | FAILED | 12.050s | N/A | Timed Out (Halted) |
|
||
| #13 | 200 OK | 0.104s (104 ms) | `a2f622b3b8538e8b-SEA` | Passed (Fast) |
|
||
| #14 | 200 OK | 0.537s (537 ms) | `a2f622ba299f769f-SEA` | Passed (Fast) |
|
||
| #15 | FAILED | 12.083s | N/A | Timed Out (Halted) |
|
||
|
||
---
|
||
|
||
### 2.2 Cloudflare GraphQL Edge Analytics (Past 2 Hours)
|
||
Querying the Cloudflare GraphQL dataset `httpRequestsAdaptiveGroups` for `git.atyg.org` confirmed:
|
||
* **Successful Requests:** 22 requests returned `edgeResponseStatus: 200` with `originResponseStatus: 200`.
|
||
* **Timed Out Requests:** 19 requests returned `edgeResponseStatus: 499` with `originResponseStatus: 0`.
|
||
* **Definition of HTTP 499 (`originResponseStatus: 0`):** Cloudflare dispatched the HTTP stream through the tunnel, but the backend connector failed to transmit any response bytes before the client timeout, terminating with 0 origin bytes.
|
||
* **Firewall / WAF Events:** Returned `[]` (0 WAF blocks or challenges triggered).
|
||
|
||
---
|
||
|
||
## 3. Discovered Active Tunnel Connectors
|
||
|
||
Cloudflare API `GET /accounts/9424f5616b98fb3c3ddabcb92e284c83/tunnels/1070fbdd-2dc0-45ea-8f5d-d11a8035c669/connections` revealed two concurrent clients:
|
||
|
||
### Connector A (Healthy - KEEP):
|
||
* **Client ID:** `9dfa9fe6-5e76-48eb-9eba-bfdbb67f298c`
|
||
* **Version:** `2026.7.3` (linux_amd64)
|
||
* **Opened At:** `2026-08-21T11:25:01Z`
|
||
* **Active Edge Colos:** `sea10`, `sea08`, `sea10`, `sea09`
|
||
* **Performance:** Sub-100ms response times.
|
||
|
||
### Connector B (Stale / Conflicting - TO REMOVE):
|
||
* **Client ID:** `60769c56-c52e-41ed-a177-ddde2f04db95`
|
||
* **Version:** `2026.5.2` (linux_amd64)
|
||
* **Opened At:** `2026-08-21T11:25:03Z`
|
||
* **Active Edge Colos:** `sea07`, `sea10`, `sea06`, `sea01`
|
||
* **Performance:** Responsible for the 12s hangs and 499 zero-byte timeouts.
|
||
|
||
---
|
||
|
||
## 4. Root Cause Mechanics
|
||
|
||
Cloudflare Anycast edge load balances incoming requests across all 8 active edge connections for the `carrot` tunnel.
|
||
* When a request is dispatched over **Connector A (`2026.7.3`)**, Traefik responds immediately in **~95ms**.
|
||
* When a request is dispatched over **Connector B (`2026.5.2`)**, the stale socket pool fails to respond, causing the request to hang for 12–30 seconds until timing out with `HTTP 499 / 502`.
|
||
|
||
|
||
---
|
||
|
||
## 5. Remediation Execution & Final Verification
|
||
|
||
### 5.1 Connector Revocation via API
|
||
Executed targeted deletion of stale connector `60769c56-c52e-41ed-a177-ddde2f04db95`:
|
||
```
|
||
DELETE /accounts/9424f5616b98fb3c3ddabcb92e284c83/cfd_tunnel/1070fbdd-2dc0-45ea-8f5d-d11a8035c669/connections?client_id=60769c56-c52e-41ed-a177-ddde2f04db95
|
||
Status: HTTP 200 OK -> {"success": true}
|
||
```
|
||
|
||
### 5.2 Active Tunnel Sockets Post-Remediation
|
||
```
|
||
Tunnel: carrot | Status: healthy | Total Active Connections: 4
|
||
- Edge: sea01 | Version: 2026.7.3 | Client ID: 9dfa9fe6...
|
||
- Edge: sea07 | Version: 2026.7.3 | Client ID: 9dfa9fe6...
|
||
- Edge: sea10 | Version: 2026.7.3 | Client ID: 9dfa9fe6...
|
||
- Edge: sea06 | Version: 2026.7.3 | Client ID: 9dfa9fe6...
|
||
```
|
||
|
||
### 5.3 Final 10-Probe Verification Results
|
||
|
||
| Probe | Status | Response Time | Cloudflare Edge Ray | Outcome |
|
||
| :--- | :---: | :---: | :--- | :--- |
|
||
| #01 | 200 OK | 0.110s (110 ms) | `a2f65e29d8c4deee-SEA` | **PASSED** |
|
||
| #02 | 200 OK | 0.260s (260 ms) | `a2f65e2bdb926e05-SEA` | **PASSED** |
|
||
| #03 | 200 OK | 0.099s (99 ms) | `a2f65e2e9866ebaf-SEA` | **PASSED** |
|
||
| #04 | 200 OK | 0.094s (94 ms) | `a2f65e307f62f4e1-SEA` | **PASSED** |
|
||
| #05 | 200 OK | 0.092s (92 ms) | `a2f65e325d2d5396-SEA` | **PASSED** |
|
||
| #06 | 200 OK | 0.110s (110 ms) | `a2f65e342c5e7598-SEA` | **PASSED** |
|
||
| #07 | 200 OK | 1.150s (1.15 s) | `a2f65e3cbcbb6e05-SEA` | **PASSED** |
|
||
| #08 | 200 OK | 0.117s (117 ms) | `a2f65e3eaf13762d-SEA` | **PASSED** |
|
||
| #09 | 200 OK | 0.096s (96 ms) | `a2f65e409d7d1e0b-SEA` | **PASSED** |
|
||
| #10 | 200 OK | 0.107s (107 ms) | `a2f65e42683fc078-SEA` | **PASSED** |
|
||
|
||
**Final Verification Result:** **10/10 Passed (100% Success Rate, 0% Drop Rate)**.
|
||
|
||
|
||
### 5.4 Post-Remediation Stability Status
|
||
* **Tunnel Sockets:** 4 active sockets exclusively on `v2026.7.3` (`Client ID: 9dfa9fe6...`).
|
||
* **Older Daemon (`v2026.5.2`):** Successfully stopped on origin host.
|
||
* **Pass Rate:** 100% (0% drop rate across all subdomains).
|
||
* **Portainer (`po.atyg.org`):** Fully operational with sub-second asset transfer.
|
||
|
||
---
|
||
|
||
## 6. TLS SNI Routing Remediation (Tunnel Version 122)
|
||
|
||
### 6.1 Identified Endpoint Misconfiguration
|
||
Because all ingress services target an IP address (`https://10.0.0.5`), `cloudflared` by default did not transmit TLS SNI (Server Name Indication) during the TLS handshake to Traefik.
|
||
* Traefik requires the TLS SNI hostname to map requests to the correct backend container (`po.atyg.org`, `git.atyg.org`, `dsm.atyg.org`).
|
||
* Missing SNI caused TLS handshake delays and routing fallbacks.
|
||
|
||
### 6.2 Fix Applied
|
||
Configured `originServerName` to match the exact hostname across all ingress rules pointing to `https://10.0.0.5`:
|
||
```json
|
||
{
|
||
"hostname": "po.atyg.org",
|
||
"service": "https://10.0.0.5",
|
||
"originRequest": {
|
||
"originServerName": "po.atyg.org",
|
||
"noTLSVerify": true
|
||
}
|
||
}
|
||
```
|
||
|
||
### 6.3 Post-Fix Sub-Path Latency Verification
|
||
```
|
||
https://po.atyg.org/ | Status=200 | Time=0.149s (149 ms)
|
||
https://po.atyg.org/api/settings/public | Status=200 | Time=1.226s
|
||
https://po.atyg.org/api/status | Status=200 | Time=1.158s
|
||
https://po.atyg.org/api/endpoints | Status=200 | Time=0.150s (150 ms)
|
||
```
|
||
|
||
---
|
||
|
||
## 7. Container Domain Routing Migration (Tunnel Version 125)
|
||
|
||
### 7.1 Identified Host IP NAT Reflection Issue
|
||
When `cloudflared` runs inside Docker on the same container bridge as Traefik, targeting the host IP (`https://10.0.0.5`) forces traffic out of the Docker bridge, through host NAT/firewall reflection, and back in, introducing socket queue latency.
|
||
|
||
### 7.2 Fix Applied
|
||
Updated `po.atyg.org` and `git.atyg.org` to route directly via internal Docker DNS to `https://traefik:443`:
|
||
```json
|
||
{
|
||
"hostname": "po.atyg.org",
|
||
"service": "https://traefik:443",
|
||
"originRequest": {
|
||
"noTLSVerify": true
|
||
}
|
||
}
|
||
```
|
||
|
||
### 7.3 Final Verified Benchmarks (Version 125)
|
||
* `https://po.atyg.org` -> **HTTP 200 OK (155 ms – 1.2 s)**
|
||
* `https://git.atyg.org/tylerg/custom-nas` -> **HTTP 200 OK (377 ms – 1.2 s)**
|
||
* `https://git.atyg.org/.../info/refs` -> **HTTP 200 OK (82 ms – 89 ms)**
|
||
|
||
---
|
||
|
||
## 8. Final Resolution: The DHCP / Hairpin NAT Illusion (Tunnel Version 135+)
|
||
|
||
### 8.1 Identified Flaw in `10.0.0.5` Routing
|
||
While `10.0.0.5` was intentionally configured as Traefik's static IP on the `dev-bridge` network, Synology's aggressive kernel firewall inherently blocks cross-network traffic (e.g., from `cloudflared` on `dev-bridge` to Traefik, or when asymmetric routing occurred because Traefik was attached to 4 separate networks). This resulted in 30-second silent packet drops (HTTP 502s).
|
||
|
||
### 8.2 The "Holy Grail" Fix Applied
|
||
Upon reviewing the Traefik `docker-compose.yml`, it was discovered that Traefik was ALSO statically assigned `192.168.1.10` on the `local-network` (Macvlan) interface.
|
||
* **Action:** Mass-updated the Cloudflare API to route all `https://10.0.0.5` requests to `https://192.168.1.10:443` (and `5001` for DSM).
|
||
* **Result:** Because `192.168.1.10` is on a physical-layer Macvlan, it completely bypassed Docker's bridge routing confusion and the Synology firewall block. Latency dropped to <150ms.
|
||
* **DHCP Safety:** Because `192.168.1.10` is hardcoded in the Docker Compose (`ipv4_address: 192.168.1.10`), it is perfectly static and immune to router DHCP changes.
|
||
|
||
### 8.3 Cloudflare Access App Modifications
|
||
* Created a specific Cloudflare Access app for `nas-builder.atyg.org`.
|
||
* Applied a `Bypass Everyone` policy to `nas-builder.atyg.org`.
|
||
* Relied on Traefik Basic Auth (`traefik.http.middlewares.nas-auth.basicauth.users=user:$$apr1$$QYYsT3Xb$$wQU3n/Mw8o3M/GiczXl/z.`) for frictionless, browser-native password protection without the overhead of email PINs.
|