All addresses, interface names, and identifiers in this article are illustrative. Public addresses use the ranges reserved for documentation in RFC 5737. No customer or internal ZSoftly values are published here.
Migration guides usually describe the move as a packaging problem. Copy the containers, copy the configuration, point DNS at the new address, and the service comes back.
We moved a VPN control plane from AWS to private cloud infrastructure. The container definitions were identical to the ones already running in production on EC2. Every service started. The host was healthy. SSH answered from the public internet.
HTTPS did not work, and it took an afternoon to find out why.
The cause was not the software. It was an assumption about the network that the public cloud had been quietly satisfying for us. That assumption is the part nobody writes down, and it is the reason this post exists.
The symptom that sends you the wrong way
The service failed in the least helpful way possible. It failed partially.
| Path | Result |
|---|---|
| SSH from the internet, port 22 | Worked |
| HTTPS from the internet, port 443 | Timed out |
| HTTP from the internet, port 80 | Timed out |
| Certificate issuance | Failed every attempt |
The certificate authority reported Timeout during connect (likely firewall problem).
Read those rows the way we first read them. Port 22 works, so the public address is live, the routing is fine, and the physical path is good. Ports 80 and 443 do not work. Something must be filtering by port, and the only things that filter by port live upstream.
That reasoning is wrong, and it is wrong in a way that costs hours. We went looking at the firewall, the switch, the VLAN, the ARP tables, and the upstream provider. All of them were healthy. One engineer power-cycled a network interface and rebooted the host on the theory that the public address had not attached correctly.
Partial failure is more expensive than total failure. Total failure tells you the path is broken. Partial failure invites you to build a theory that explains only half the evidence.
What the packets said
A packet capture on the public interface of the server itself ended the guessing.
198.51.100.20 > 203.0.113.10.443: Flags [S]
198.51.100.20 > 203.0.113.10.443: Flags [S]
198.51.100.20 > 203.0.113.10.80: Flags [S]
...That is the certificate authority’s validation server retrying, seven times per port, and reaching us. Every packet arrived. Nothing upstream was blocking anything.
There were no replies. The connection requests landed on the server and the server never answered.
This single observation eliminated the firewall, the switch, the upstream provider, and the hypervisor in one step. It also invalidated the diagnosis we had been working from for most of the afternoon. The lesson is not subtle. Before theorizing about who is blocking traffic, confirm whether the traffic arrives.
The assumption the public cloud was hiding
Our EC2 instance had one network interface. The provider mapped the public address to it. Traffic arrived, replies left, and nobody on the team thought about return paths.
Our private cloud server has two interfaces. One carries the private network and holds the default route. The other carries the public address. Because the default route points at the private side, replies from the public address need an explicit rule to leave through the correct interface:
ip rule 100: from 203.0.113.10 lookup 200That rule matches on source address. It is correct, it is standard practice, and it works for services running on the host. SSH replies originate from the public address, the rule matches, and the packet leaves through the public interface. That is why SSH worked the entire time.
Published container ports behave differently. Traffic to a published port is translated and forwarded to the container. The reply is routed while its source is still the container address, before the translation is reversed. The rule matches on the public address. The container address is not the public address, so the rule does not match. The reply falls through to the default route and leaves through the private interface, where it disappears.
The kernel confirms both behaviors directly:
reply to a neighbor on the same subnet -> dev eth1 correct
reply to a client on the internet -> via 10.0.0.1 dev eth0 wrong interfaceEvery service on the host worked. Every service in a container did not. Nothing in the container configuration was different from the version running in AWS, because the difference was never in the containers.
The test that lies to you
Partway through, we tested from a neighboring server on the same network segment. It returned HTTP 308 in three milliseconds. The service looked healthy.
That test was worthless, and worse than worthless because it was convincing.
Replies to a machine on the same subnet do not need the default route. They resolve through a directly connected route that exists regardless of the policy rule. The broken path is only the one that requires the default route, which means only clients on the internet experience the failure. A same-subnet test passes whether the bug is present or not.
Any validation that runs inside your own network reproduces this class of false pass. If clients on the internet have to reach a service, test it from the internet.
The fix
Instead of routing on source address, tag the connection when it arrives on the public interface and route the reply by that tag.
iptables -t mangle -A PREROUTING -i <public> -m conntrack --ctstate NEW \
-j CONNMARK --set-xmark 0x200/0xffff
iptables -t mangle -A PREROUTING ! -i <public> -m conntrack --ctstate ESTABLISHED,RELATED \
-j CONNMARK --restore-mark --nfmask 0xffff --ctmask 0xffff
ip rule add fwmark 0x200/0xffff lookup 200 priority 101The connection mark survives address translation. The kernel routes the reply correctly, whatever the source address looks like when it makes the routing decision.
Note the second line. Restoring the mark on every interface also marks inbound packets, which then follow the same routing table. That table has no route to the container network, so the inbound packet matches its default route and goes straight back out to the internet. We made exactly that mistake, and the packet capture showed the internal container address leaving through the public interface. Restricting the restore to non-public interfaces fixes it.
Choose the mark value with care. Check which packet marks other software on the host already uses, then pick a value and mask that do not overlap.
Error messages are evidence
After the first version of the fix, the certificate authority changed its complaint from Timeout during connect to Timeout after connect.
Those look like the same message. They are not.
| Message | Meaning |
|---|---|
Timeout during connect |
The connection was never established. Nothing answered. |
Timeout after connect |
The connection was established. The response never completed. |
That one word confirmed the reply path was now working and moved the investigation to a different stage of the same connection. Reading the exact wording saved another round of guessing.
Use the staging environment
Certificate authorities limit failed validations per hostname per hour. A migration that fails repeatedly exhausts that limit and leaves you waiting instead of working.
A staging environment has far higher limits and exercises the identical network path: the same DNS lookup, the same inbound connection, the same challenge response. It proves everything except trust.
We validated against staging until the challenge succeeded, then switched to production and issued a trusted certificate on the first attempt. Do not treat a staging certificate as a finished service. It proves the path works. It does not mean a browser will accept it.
Protect your access paths before you test
One more lesson, learned the uncomfortable way.
During the investigation, we stopped a router to eliminate it as a variable. That router happened to carry the private network path used for administration. Access to every other server in the region disappeared at the same moment, in the middle of an incident.
The service being migrated was unaffected, because it had its own public address and direct access. That was luck rather than design.
Before removing anything from a live path, write down three things. How you reach each system. Which of those paths depends on the thing you are about to stop. What remains if you are wrong. A migration is exactly the period when your normal access assumptions are least reliable.
Everything here was managed as code
We applied nothing by hand on the server, including the fix.
The public routing rule, the connection marking, the systemd unit that reapplies it after a reboot, the certificate configuration, and the container definitions are all Ansible. The investigation used a live host, but the remedy went in as a reviewed change to a role, applied through the same automation that built the machine.
That mattered twice. The first version of the fix was wrong. Rolling it forward was a one-line edit and a rerun, not an undocumented change with no record for the next person. When the second router is rebuilt, it inherits the corrected behavior without anyone remembering this incident.
Anything applied by hand during an incident is a defect waiting to reappear on the next host. If a fix is worth keeping, it belongs in the automation before the incident is closed.
We split the work across three people: the migration off AWS, the cutover and configuration on the private cloud side, and an independent review and QA pass. The review is the part teams skip under time pressure, and it is the part that caught the difference between a staging certificate and a working service.
What to check before a public-to-private cloud migration
The application layer is rarely the hard part. These are the questions that decide whether a working public-cloud deployment survives the move:
- How many network interfaces does the destination host have, and which one holds the default route?
- Does the public address require a policy routing rule, and does that rule match on source address?
- Does any workload publish ports through address translation, such as a container runtime?
- How do you validate the return path, and does that test originate outside your own network?
- Which assumptions did the public cloud satisfy silently, including address mapping, return routing, and interface count?
- What is the administrative access path to each system, and what breaks it?
- Which certificate authority limits apply, and is there a staging environment to absorb the failures?
- Does the deployment reference implementation encode environment assumptions that no longer hold?
Question eight is the one that generalizes. A reference implementation proven in one environment carries that environment’s assumptions with it. Our container definitions were correct. They had never needed to describe a return path, because in the public cloud there was only one way out.
The takeaway
Moving a workload to private cloud infrastructure means taking ownership of the things the public cloud did for you silently, with no invoice line and no configuration entry. Single-interface networking and automatic return routing are two of them.
The software was never wrong. The environment changed, and one routing rule that had always been sufficient stopped covering half the traffic.
When a service works on one port and fails on another, resist the conclusion that something upstream is filtering. Capture the traffic first. If the packets arrive and nothing answers, the problem is yours, and it is usually on the host, not the firewall.
Talk to us
We build and operate self-hosted zero trust networking on WireGuard, including migrations off legacy VPN appliances and per-user subscriptions. The control server runs on infrastructure you own.
Planning a move like this? Staring at a service that answers on one port and times out on another? We have already spent the afternoon on it.
- Zero Trust VPN with Headscale, our implementation and managed service
- Contact us to talk through a migration or an access design