When the Root Cause Never Gets a Name: The Portainer Phantom Timeout

TL;DR: Portainer's UI became completely unreachable from a management machine while every server-side signal said the service was healthy — correct port binding, a correct firewall rule, packet counters proving inbound connections were actually arriving and being forwarded. A reboot fixed it instantly. The honest answer is that this one was never conclusively diagnosed — and the right response wasn't to keep chasing an explanation, it was to build automated detection and recovery around a real but rare failure mode.

The symptom

Portainer's web UI stopped responding from a Windows management machine. Every server-side indicator looked completely normal: the port was bound correctly (0.0.0.0:9443), the UFW rule allowing it was present and correct, and curl run directly from the VM's own console succeeded without issue.

Ruling things out, one layer at a time

1. Checked Docker's NAT rules directly, not just whether the port was open: sudo iptables -t nat -L -n -v | grep 9443 The DNAT rule showed a non-zero packet counter — meaning inbound connection attempts genuinely were reaching the VM and being correctly forwarded into the container. This ruled out the VM's network stack and Docker's port mapping in one check, not just made them look unlikely.
2. Used real curl.exe, not PowerShell's Invoke-WebRequest alias, from the Windows machine — confirmed a genuine timeout, not something specific to a browser or a scripting shim.
3. Compared against a working port from the same client: Test-NetConnection against port 22 (SSH) succeeded from the identical machine that was failing against port 9443, ruling out any blanket network path problem between client and VM.
4. Checked Proxmox's firewall, both levels — disabled, ruled out.
5. Checked Windows Firewall outbound rules, then disabled it entirely as a test — no change.
6. Rebooted the Portainer VM. The problem disappeared immediately.

Root cause

Never actually identified. Every layer of configuration checked out clean, and the one piece of hard evidence available — the DNAT packet counter — specifically showed traffic arriving and being routed correctly, while new external TCP connections still weren't completing. That combination points toward a stuck kernel-level network state on the guest, a known but notoriously hard-to-pin-down class of issue in virtualized Linux guests — but the reboot fixing it supports that read without actually proving it. It's worth being precise about that distinction rather than writing a confident root cause that isn't really earned by the evidence.

Fix

Immediate: reboot the VM.

Durable: an automated external health check, run from the Ansible control node rather than from the affected VM itself specifically so it tests the same path a real user actually takes — curling the service on a timer, triggering an automatic reboot after three consecutive failures.

What this demonstrates

Not every bug resolves to a root cause you can name, and that's worth saying plainly rather than writing a more confident-sounding conclusion than the evidence actually supports. When every layer of configuration checks out clean and a restart silently fixes it, the right engineering response isn't indefinite continued investigation looking for an explanation that may not be findable with the evidence available — it's accepting the failure mode as real but rare, and building automated detection and recovery around it. The health check here isn't a placeholder for a "real" fix that never happened; it's a deliberate, permanent mitigation for a bug that genuinely couldn't be conclusively diagnosed with the tools available at the time.