MSK · UTC+3 · 18 Nov 2026one engagement open · Dec–Jan

Work

Four jobs, described the way they actually went

Renumbering a 14-node edge without downtime

A hosting provider had to return a /24 to its upstream. Fourteen machines, six of them carrying customer traffic, and a strict window: the upstream switched routes on Monday morning.

We moved addresses in three passes. First the new addresses were added alongside the old ones and tested from outside the network. Then DNS TTLs were dropped to 60 seconds a week in advance. On the night, each node was renumbered, its certificates re-issued, and its monitoring re-pointed — one node at a time, with a five-minute pause to watch the graphs.

old /24 active new /24 active cutover night
Both blocks lived in parallel for a week; the old one was returned only after the graphs were flat.

The only visible change for customers was a routing table that suddenly had better neighbours. The three things that saved the night: parallel addressing, TTLs lowered in advance, and one engineer reading graphs while the other typed.

Backup restore: 6 hours to 40 minutes

A media archive of 1.2 TB was being backed up nightly with rsync to a single external disk. The restore was never tested. When we tried it, a full restore took six hours and failed twice on permission errors.

We moved to restic with a local cache and a second copy off-site. Retention was set to 7 daily, 4 weekly and 6 monthly snapshots — about 2.3 TB of storage for a 1.2 TB dataset, which was acceptable. The first restore drill took 52 minutes; after adding the cache, 40.

before — 6 h 05 after — 40 min full restore of 1.2 TB, measured on the same hardware
Same dataset, same disk, same network. The difference is a local cache and a restore that is rehearsed.

A nine-month packet loss bug

Customers on one link complained about "slow evenings" since spring. Ping was perfect. Speed tests were perfect. Every ordinary check agreed that nothing was wrong.

Two days of captures showed TCP retransmissions with a tiny advertised MSS and lost full-size segments on one transit link. The link's MTU was 1400; the routers on both sides assumed 1500 and filtered the ICMP that would have said so. Clamping MSS on the two border routers ended a nine-month ticket in one line of nftables.

The certificate runbook

A renewal missed silently on a Saturday: certbot succeeded on twelve hosts and failed on one, and nobody was watching that one because it was "the boring host". Two weeks later, mobile apps started refusing connections.

The fix was not monitoring — it was a runbook: renewals go through one script, the script prints a single line per host, and the line is checked by a human every Monday. Boring, written down, and still in use by a team of four.