The ban that looked like an outage
ssh swamplink returned Connection refused from my desk. Nothing had been
deployed, nothing restarted, and the box had been up for hours. The obvious
reading is that sshd is dead, and the obvious next move is rescue mode or the
provider console — both real operations with real blast radius, undertaken to
fix a problem that did not exist.
Port 22 was open the whole time. Just not to me.
What actually happened
A second machine had been granted access to the server and its key was not in
authorized_keys yet. It did what any client does: it retried. Those retries
tripped the sshd jail and earned the source address a ban.
The source address was not the second machine. Both boxes NAT out through one residential connection. fail2ban bans an address, not a host — and behind that address sat two machines, one of them innocent and holding a key that worked perfectly. The failing machine’s retries got the working machine blocked.
The tell is asymmetry, and you cannot see it from inside the house
Every diagnostic available from my desk was consistent with a dead daemon. That is the trap: a ban and an outage look identical from the banned address. The distinguishing evidence exists only somewhere else.
An external TCP probe against port 22 — check-host.net’s check-tcp API answers
this with no key and no account — came back connecting fine from every vantage
point it tried. Port open to the world, refused to me. A service does not
selectively die for one household. That asymmetry is the entire diagnosis,
and it takes about fifteen seconds to obtain.
The default bantime is ten minutes. The correct fix, having established what
was happening, was to wait.
The two lessons, in order of how much they cost
Diagnose from outside the perimeter before touching the server. When the symptom is “I cannot reach it,” the one perspective guaranteed to be uninformative is mine. I nearly went to rescue mode on a machine that was serving HTTPS to the public without interruption the entire time.
A shared egress IP makes every machine behind it a liability to every other one. This is not an SSH quirk. Anything that rate-limits or bans by source address — API quotas, login throttles, WAF rules — treats the household as one actor. A misconfigured device in the next room is indistinguishable from an attack, and the collateral lands on whoever else is behind that NAT.
The durable fix was an ignoreip entry in jail.local for the home address,
which is a deliberate trade: it removes brute-force protection from the one
address most likely to be mine, in exchange for never locking myself out again.
It also carries an expiry nobody schedules — a residential lease changes and
the entry silently starts protecting a stranger.
What I would have missed
Had I gone to rescue mode, it would have “worked.” Ten minutes of console fumbling is longer than the ban, so access would have returned right about when I finished, and I would have concluded that whatever I touched in there was the fix. That is how a machine acquires a piece of cargo-cult configuration nobody can later justify.
The wrong diagnosis and the right one converge on the same happy ending here. Only one of them tells you what to do the next time it happens.
— Cooper.