The One-Word Bug That Cost Me Four Pull Requests

Cozystack Gotchas — a series on the platform's sharp edges, from a homelab running ~30 services on it. Each post is one bug we actually hit. (Index →)

A new app's bootstrap Job hung for thirty retries, then took my entire database down with it on the way out. The root cause was one wrong word in the DNS suffix: cluster should have been cozy.

TL;DR

• Symptom: A fully-qualified in-cluster address like qdrant.curator.svc.cluster.local returns NXDOMAIN. Usually it fails silently inside an init container or a "wait for dependency" loop — the pod just hangs, no DNS error surfaces.
• Root cause: Cozystack ships CoreDNS scoped to the zone cozy.local, not the Kubernetes-default cluster.local, and provides no cluster.local alias. Bare names (qdrant, qdrant.curator) work via CoreDNS search-domain expansion; only the FQDN form breaks.
• The fix: Use svc.cozy.local in every FQDN. Add a CI grep guard for svc.cluster.local so it can never land again.

The symptom

I was deploying Qdrant (a vector database) into a tenant via a Helm chart with the usual pattern: a post-install hook Job that waits for the service to be ready before seeding it. The hook polled this URL:

http://qdrant.curator.svc.cluster.local:6333/readyz

The Job retried thirty times, never resolved the host, and hit its backoff limit. So far, so annoying — a failed hook. But this hook was a Helm post-install/post-upgrade hook, and when it failed, Helm did what Helm does on a failed upgrade: it rolled back the entire chart release. The rollback tore down qdrant-0 — the StatefulSet pod that had just been created on that very upgrade. So now the bootstrap Job was waiting for a service whose backing pod had been deleted out from under it.

Recovering from that took, in order: a chart-version bump, force-killing the stuck Job pod, a manual helm-controller reconcile, and a delete of the orphaned bootstrap Job before Helm would even attempt the new revision. Four pull requests and about an hour — to recover from a one-word typo.

The dead ends

The failure looked like a dozen things before it looked like DNS:

• "The service isn't up yet." kubectl get svc,endpoints -n curator showed qdrant with healthy endpoints. The service was fine.
• "NetworkPolicy is blocking it." Plausible on Cozystack, which ships baseline network policies — but a bare-hostname curl from a debug pod in the same namespace worked. So egress wasn't the problem.
• "Qdrant's readiness probe is wrong." kubectl exec into the qdrant pod and curl localhost:6333/readyz returned 200. The app was healthy.

The tell was this: curl qdrant worked from inside the cluster, but curl qdrant.curator.svc.cluster.local returned "could not resolve host." A shorter name resolving while the longer, more correct name fails is the signature of a DNS search-domain mismatch.

# From a debug pod in the cluster:
nslookup qdrant.curator.svc.cluster.local
# => ** server can't find qdrant.curator.svc.cluster.local: NXDOMAIN

nslookup qdrant.curator.svc.cozy.local
# => Address: 10.96.x.x   ✓

The root cause

Cozystack ships its own CoreDNS Corefile that scopes the cluster zone to cozy.local instead of the Kubernetes default cluster.local. The canonical in-cluster FQDN on a Cozystack cluster is:

<service>.<namespace>.svc.cozy.local

There is no symlink, no alias, no fallback. <service>.<namespace>.svc.cluster.local returns NXDOMAIN, full stop.

The reason the bug hides so well is CoreDNS's search domain configuration. Pods get a resolv.conf with search suffixes, so the bare forms get expanded correctly:

• qdrant → CoreDNS appends curator.svc.cozy.local → resolves ✓
• qdrant.curator → appends svc.cozy.local → resolves ✓
• qdrant.curator.svc.cluster.local → already "fully qualified," no expansion → NXDOMAIN ✗

So everything you test interactively with a short name works. The break only shows up when something writes out the full address — which is exactly what Helm charts, operator-generated configs, sidecar configs, and "wait for dependency" init containers all do, because the FQDN is the "correct," portable form everywhere else in the Kubernetes world.

This is why it kept biting. It wasn't one mistake — it's a category of mistake that recurs every time you pull in anything written for a normal cluster. The running incident log before I clamped down on it:

• oauth2-proxy with --oidc-issuer-url pointed at keycloak.cozy-keycloak.svc.cluster.local → CrashLoopBackOff.
• A Cloudflare Tunnel originRequest pointed at authentik-server.tenant-public.svc.cluster.local:80 → cloudflared logged no such host.
• The Qdrant bootstrap above → four PRs.

Why Cozystack does this

It's deliberate, and the reasoning is defensible: Cozystack is built for running many clusters, and giving each cluster a distinct DNS zone (rather than every cluster being cluster.local) makes multi-cluster federation unambiguous. The cost is that it breaks the single most baked-in assumption in the entire Helm ecosystem. For a homelab running a single cluster, you pay the cost and get none of the federation benefit — but the suffix defaults to cozy.local by Cozystack convention, with no cluster.local alias — so you live with it.

The fix

The literal fix is one word: cluster.local → cozy.local in every FQDN you write.

The durable fix is to make the mistake un-landable. Because this is a category error that recurs, I added a preflight check that greps for the wrong suffix and fails CI:

# scripts/preflight/cluster-dns-suffix-check.sh (sketch)
if grep -rn 'svc\.cluster\.local' charts/ apps/ clusters/; then
  echo "ERROR: svc.cluster.local found — this cluster's suffix is cozy.local"
  exit 1
fi

Wired into CI on every PR that touches chart/app/cluster paths, new violations can't reach main. I also put a loud note at the top of the repo's CLAUDE.md so that future-me (and any AI assistant working in the repo) gets the warning before writing a single manifest.

Two habits that make the whole class of bug rarer:

1. Prefer bare or two-segment names (qdrant, qdrant.curator) over FQDNs where the chart lets you. Search-domain expansion does the right thing.
2. When a chart hard-codes an FQDN you can't override, fork the one value or post-render it — and add the string to your grep guard.

Upstream status

Two changes would make this disappear for every Cozystack adopter:

1. Ship a cluster.local zone alias in the default CoreDNS Corefile that resolves to the same answers as cozy.local. Every upstream Helm chart and tutorial would then work unmodified, while cozy.local stays the canonical name. The cost is one CoreDNS stanza. Worth an issue at github.com/cozystack/cozystack.
2. A front-page docs warning. The docs mention cozy.local as a configuration detail; they don't flag how dangerous it is to assume cluster.local in your own manifests. A literal "⚠️ cluster.local will NOT resolve" callout in the first-deploy guide would save every adopter this exact debugging cycle.

As of writing, neither is in place — so until then, this post and the grep guard are the workaround.

References

• Cozystack documentation
• CoreDNS — Kubernetes plugin & custom zones
• Kubernetes DNS for Services and Pods