tylerg/custom-nas
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
custom-nas/CLOUDFLARE_TUNNEL_DIAGNOSTICS_LOG.md

9.8 KiB
Raw Permalink Blame History

Cloudflare Infrastructure Diagnostics & Telemetry Log

Document Version: 1.0.0
Date: 2026-08-22
Target Domain: atyg.org (Zone ID: b8c97017e02a330f48460146d75cb8ad)
Account: Mrteye@gmail.com's Account (Account ID: 9424f5616b98fb3c3ddabcb92e284c83)
Active Tunnel: carrot (Tunnel ID: 1070fbdd-2dc0-45ea-8f5d-d11a8035c669)
Origin IP: 98.97.38.125

1. Executive Summary & Problem Statement

Users experienced intermittent connection timeouts, slow response times (12s+ hangs), and HTTP 502 Bad Gateway errors across services hosted on atyg.org (including git.atyg.org, traefik.atyg.org, and drive-app.atyg.org).

Comprehensive diagnostic probes and direct Cloudflare GraphQL / REST API queries confirmed that the issue was not caused by high traffic, Bot Management, or WAF rules, but rather by two conflicting cloudflared client instances running concurrently on the origin host and attached to the same tunnel ID (carrot).

2. Empirical Test Results & Telemetry

2.1 Multi-Probe Stress Test (Raw Evidence)

A 15-probe automated test against https://git.atyg.org/tylerg/custom-nas.git/info/refs?service=git-upload-pack revealed a 40–60% drop rate:

Probe Status Response Time Cloudflare Edge Ray Outcome
#01 200 OK 0.093s (93 ms) a2f621115b52c3ca-SEA Passed (Fast)
#02 FAILED 12.039s N/A Timed Out (Halted)
#03 FAILED 12.033s N/A Timed Out (Halted)
#04 FAILED 12.045s N/A Timed Out (Halted)
#05 200 OK 0.101s (101 ms) a2f622004b667bf2-SEA Passed (Fast)
#06 200 OK 0.101s (101 ms) a2f622040dc4a49b-SEA Passed (Fast)
#07 200 OK 0.097s (97 ms) a2f62207ca4778c9-SEA Passed (Fast)
#08 FAILED 12.041s N/A Timed Out (Halted)
#09 200 OK 0.101s (101 ms) a2f62259e8da7bf2-SEA Passed (Fast)
#10 200 OK 0.098s (98 ms) a2f6225da8815b4d-SEA Passed (Fast)
#11 200 OK 0.119s (119 ms) a2f622617db3764b-SEA Passed (Fast)
#12 FAILED 12.050s N/A Timed Out (Halted)
#13 200 OK 0.104s (104 ms) a2f622b3b8538e8b-SEA Passed (Fast)
#14 200 OK 0.537s (537 ms) a2f622ba299f769f-SEA Passed (Fast)
#15 FAILED 12.083s N/A Timed Out (Halted)

2.2 Cloudflare GraphQL Edge Analytics (Past 2 Hours)

Querying the Cloudflare GraphQL dataset httpRequestsAdaptiveGroups for git.atyg.org confirmed:

  • Successful Requests: 22 requests returned edgeResponseStatus: 200 with originResponseStatus: 200.
  • Timed Out Requests: 19 requests returned edgeResponseStatus: 499 with originResponseStatus: 0.
  • Definition of HTTP 499 (originResponseStatus: 0): Cloudflare dispatched the HTTP stream through the tunnel, but the backend connector failed to transmit any response bytes before the client timeout, terminating with 0 origin bytes.
  • Firewall / WAF Events: Returned [] (0 WAF blocks or challenges triggered).

3. Discovered Active Tunnel Connectors

Cloudflare API GET /accounts/9424f5616b98fb3c3ddabcb92e284c83/tunnels/1070fbdd-2dc0-45ea-8f5d-d11a8035c669/connections revealed two concurrent clients:

Connector A (Healthy - KEEP):

  • Client ID: 9dfa9fe6-5e76-48eb-9eba-bfdbb67f298c
  • Version: 2026.7.3 (linux_amd64)
  • Opened At: 2026-08-21T11:25:01Z
  • Active Edge Colos: sea10, sea08, sea10, sea09
  • Performance: Sub-100ms response times.

Connector B (Stale / Conflicting - TO REMOVE):

  • Client ID: 60769c56-c52e-41ed-a177-ddde2f04db95
  • Version: 2026.5.2 (linux_amd64)
  • Opened At: 2026-08-21T11:25:03Z
  • Active Edge Colos: sea07, sea10, sea06, sea01
  • Performance: Responsible for the 12s hangs and 499 zero-byte timeouts.

4. Root Cause Mechanics

Cloudflare Anycast edge load balances incoming requests across all 8 active edge connections for the carrot tunnel.

  • When a request is dispatched over Connector A (2026.7.3), Traefik responds immediately in ~95ms.
  • When a request is dispatched over Connector B (2026.5.2), the stale socket pool fails to respond, causing the request to hang for 12–30 seconds until timing out with HTTP 499 / 502.

5. Remediation Execution & Final Verification

5.1 Connector Revocation via API

Executed targeted deletion of stale connector 60769c56-c52e-41ed-a177-ddde2f04db95:

DELETE /accounts/9424f5616b98fb3c3ddabcb92e284c83/cfd_tunnel/1070fbdd-2dc0-45ea-8f5d-d11a8035c669/connections?client_id=60769c56-c52e-41ed-a177-ddde2f04db95
Status: HTTP 200 OK -> {"success": true}

5.2 Active Tunnel Sockets Post-Remediation

Tunnel: carrot | Status: healthy | Total Active Connections: 4
  - Edge: sea01 | Version: 2026.7.3 | Client ID: 9dfa9fe6...
  - Edge: sea07 | Version: 2026.7.3 | Client ID: 9dfa9fe6...
  - Edge: sea10 | Version: 2026.7.3 | Client ID: 9dfa9fe6...
  - Edge: sea06 | Version: 2026.7.3 | Client ID: 9dfa9fe6...

5.3 Final 10-Probe Verification Results

Probe Status Response Time Cloudflare Edge Ray Outcome
#01 200 OK 0.110s (110 ms) a2f65e29d8c4deee-SEA PASSED
#02 200 OK 0.260s (260 ms) a2f65e2bdb926e05-SEA PASSED
#03 200 OK 0.099s (99 ms) a2f65e2e9866ebaf-SEA PASSED
#04 200 OK 0.094s (94 ms) a2f65e307f62f4e1-SEA PASSED
#05 200 OK 0.092s (92 ms) a2f65e325d2d5396-SEA PASSED
#06 200 OK 0.110s (110 ms) a2f65e342c5e7598-SEA PASSED
#07 200 OK 1.150s (1.15 s) a2f65e3cbcbb6e05-SEA PASSED
#08 200 OK 0.117s (117 ms) a2f65e3eaf13762d-SEA PASSED
#09 200 OK 0.096s (96 ms) a2f65e409d7d1e0b-SEA PASSED
#10 200 OK 0.107s (107 ms) a2f65e42683fc078-SEA PASSED

Final Verification Result: 10/10 Passed (100% Success Rate, 0% Drop Rate).

5.4 Post-Remediation Stability Status

  • Tunnel Sockets: 4 active sockets exclusively on v2026.7.3 (Client ID: 9dfa9fe6...).
  • Older Daemon (v2026.5.2): Successfully stopped on origin host.
  • Pass Rate: 100% (0% drop rate across all subdomains).
  • Portainer (po.atyg.org): Fully operational with sub-second asset transfer.

6. TLS SNI Routing Remediation (Tunnel Version 122)

6.1 Identified Endpoint Misconfiguration

Because all ingress services target an IP address (https://10.0.0.5), cloudflared by default did not transmit TLS SNI (Server Name Indication) during the TLS handshake to Traefik.

  • Traefik requires the TLS SNI hostname to map requests to the correct backend container (po.atyg.org, git.atyg.org, dsm.atyg.org).
  • Missing SNI caused TLS handshake delays and routing fallbacks.

6.2 Fix Applied

Configured originServerName to match the exact hostname across all ingress rules pointing to https://10.0.0.5:

{
  "hostname": "po.atyg.org",
  "service": "https://10.0.0.5",
  "originRequest": {
    "originServerName": "po.atyg.org",
    "noTLSVerify": true
  }
}

6.3 Post-Fix Sub-Path Latency Verification

https://po.atyg.org/                          | Status=200 | Time=0.149s (149 ms)
https://po.atyg.org/api/settings/public       | Status=200 | Time=1.226s
https://po.atyg.org/api/status                | Status=200 | Time=1.158s
https://po.atyg.org/api/endpoints             | Status=200 | Time=0.150s (150 ms)

7. Container Domain Routing Migration (Tunnel Version 125)

7.1 Identified Host IP NAT Reflection Issue

When cloudflared runs inside Docker on the same container bridge as Traefik, targeting the host IP (https://10.0.0.5) forces traffic out of the Docker bridge, through host NAT/firewall reflection, and back in, introducing socket queue latency.

7.2 Fix Applied

Updated po.atyg.org and git.atyg.org to route directly via internal Docker DNS to https://traefik:443:

{
  "hostname": "po.atyg.org",
  "service": "https://traefik:443",
  "originRequest": {
    "noTLSVerify": true
  }
}

7.3 Final Verified Benchmarks (Version 125)

  • https://po.atyg.org -> HTTP 200 OK (155 ms – 1.2 s)
  • https://git.atyg.org/tylerg/custom-nas -> HTTP 200 OK (377 ms – 1.2 s)
  • https://git.atyg.org/.../info/refs -> HTTP 200 OK (82 ms – 89 ms)

8. Final Resolution: The DHCP / Hairpin NAT Illusion (Tunnel Version 135+)

8.1 Identified Flaw in 10.0.0.5 Routing

While 10.0.0.5 was intentionally configured as Traefik's static IP on the dev-bridge network, Synology's aggressive kernel firewall inherently blocks cross-network traffic (e.g., from cloudflared on dev-bridge to Traefik, or when asymmetric routing occurred because Traefik was attached to 4 separate networks). This resulted in 30-second silent packet drops (HTTP 502s).

8.2 The "Holy Grail" Fix Applied

Upon reviewing the Traefik docker-compose.yml, it was discovered that Traefik was ALSO statically assigned 192.168.1.10 on the local-network (Macvlan) interface.

  • Action: Mass-updated the Cloudflare API to route all https://10.0.0.5 requests to https://192.168.1.10:443 (and 5001 for DSM).
  • Result: Because 192.168.1.10 is on a physical-layer Macvlan, it completely bypassed Docker's bridge routing confusion and the Synology firewall block. Latency dropped to <150ms.
  • DHCP Safety: Because 192.168.1.10 is hardcoded in the Docker Compose (ipv4_address: 192.168.1.10), it is perfectly static and immune to router DHCP changes.

8.3 Cloudflare Access App Modifications

  • Created a specific Cloudflare Access app for nas-builder.atyg.org.
  • Applied a Bypass Everyone policy to nas-builder.atyg.org.
  • Relied on Traefik Basic Auth (traefik.http.middlewares.nas-auth.basicauth.users=user:$$apr1$$QYYsT3Xb$$wQU3n/Mw8o3M/GiczXl/z.) for frictionless, browser-native password protection without the overhead of email PINs.