I wanted zero trust in a cluster the way I wanted perfect crosswind landings: clearly the right idea, harder in practice than in the briefing. The first time I applied a default-deny NetworkPolicy with confidence and watched health checks fail, I was not thinking about microsegmentation anymore. I was thinking about rollback. Production traffic was fine for four minutes, then it was not, and the problem was me — not the CNI, not the cloud, not Kubernetes being mysterious.

Ethernet cables in a network switch

Photo by Pixabay on Pexels

NetworkPolicy is powerful and easy to misuse. It declares who may talk to whom at L3/L4. It does not understand your application intent. It does not know that your Pod still needs kube-dns on port 53, or that the metrics scraper lives in another namespace, or that your egress to S3 goes through a proxy you forgot to allow. The cluster enforces what you wrote. That is the feature. That is also the trap.

I am not a security architect. I am an engineer who has broken prod with good intentions and cleaned it up. This is the incremental approach I use now when someone says “we should lock down the network” and everyone nods because nobody wants to be the person who argues against security.

What NetworkPolicy actually does (and does not)

In most CNIs that support it — Calico, Cilium, OVN-Kubernetes on OpenShift, others with caveats — a NetworkPolicy selects Pods with labels and defines ingress, egress, or both. If a Pod is selected by a policy that restricts traffic, only what is explicitly allowed passes. Pods not selected by any policy typically keep default allow behavior, depending on CNI and platform defaults. That asymmetry is the first thing to verify on your cluster before you write YAML.

NetworkPolicy does not:

  • Encrypt traffic (that is mTLS, service mesh, or TLS between apps)
  • Replace authentication or authorization at the API
  • Fix vulnerable images or leaked secrets
  • Understand HTTP paths — it sees ports and protocols

It does segment the blast radius when something inside the cluster is compromised or misconfigured. A worm that cannot reach the database Pod on 5432 because egress from the compromised namespace is denied is a win. A deploy that cannot reach the database because you typoed a label selector is a postmortem.

I treat NetworkPolicy as a fence you build while still living in the house, not a wall you pour in one pour.

Why big-bang default deny fails in real teams

The textbook zero-trust diagram shows a default-deny namespace and explicit allow rules for every flow. Clean. Correct on a whiteboard. In a running platform with dozens of services, unknown cron jobs, legacy Helm charts, and a monitoring stack nobody fully documented, default deny on Monday morning is a stress test nobody agreed to take.

What breaks first is rarely the main API:

DNS. UDP and TCP port 53 to kube-system or CoreDNS endpoints. Deny egress and your Pod resolves nothing. Symptoms look like “database is down” when the database is fine.

Control plane and webhooks. Admission controllers, operators, cloud metadata endpoints — depending on architecture, some workloads need paths you did not map.

Observability. Prometheus scraping Pod metrics on high ports, log agents, traces exporters. Silence ingress from monitoring and your alerts go quiet while the app burns.

Image pulls and registries. Usually node-level, not Pod egress, until you use egress policies that accidentally block sidecars talking to local agents.

External dependencies. Payment APIs, identity providers, object storage, webhooks to SaaS. Each needs an egress rule or a shared egress gateway pattern.

The failure mode is organizational too. App teams get a policy dropped on them without a map of their dependencies. Platform teams become ticket queues for “please allow IP x.” Everyone hates NetworkPolicy. Six months later someone removes it “temporarily.” I have seen that movie.

Incremental rollout keeps trust intact.

The one-rule-at-a-time playbook

This is the sequence I follow. It is slower than a manifest that denies everything. It is faster than recovery.

Phase 0 — Know your CNI and defaults

Before writing policy, I confirm:

  • NetworkPolicy is enforced (not just accepted by the API)
  • Whether global default deny exists already
  • How DNS is exposed (CoreDNS Service IP, NodeLocal DNSCache, OpenShift dns-default)
  • Whether the mesh or egress gateway already centralizes outbound traffic

On OpenShift I also check if AdminNetworkPolicy or BaselineAdminNetworkPolicy exists at the cluster level. Platform defaults can surprise you if you only look at namespace YAML.

Phase 1 — Inventory without enforcement

I generate a picture of actual flows before I restrict anything. Options:

Flow logs from CNI — Calico flow logs, Cilium Hubble, cloud VPC flow logs if traffic leaves the node.

Service dependency maps — even a spreadsheet from team interviews beats guessing.

Staging rehearsal — same CNI, same DNS layout, policies applied there first.

I label this phase “measure twice” because the cutover is a cut.

Phase 2 — Allow observability and DNS explicitly

My first policies are boring and wide within the namespace:

  • Allow ingress from the monitoring namespace to scrape metrics ports
  • Allow egress to DNS (kube-system or cluster DNS Service) on 53/tcp and 53/udp
  • Allow egress to same-namespace Pods if the app uses headless services or sidecars

I apply these before any deny rule. Verify health checks and dashboards. If Prometheus goes quiet, fix that before proceeding.

Example egress for DNS (adjust namespace labels to your cluster):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Boring YAML is good YAML here.

Phase 3 — Default deny in one namespace, not the cluster

I pick a non-critical namespace first — internal tooling, a new service with few dependencies, something with friendly owners and good tests. I apply:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: tooling
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Then I add explicit allow rules one at a time, each merged from staging, each validated:

  1. Ingress from ingress controller namespace to app port
  2. Egress to database namespace on 5432
  3. Egress to Redis on 6379
  4. Egress to external API via CIDR or namespace of egress proxy

One rule, one deploy, one verification window. I watch application metrics, synthetic checks, and CNI drop counters if available. Roll back the last rule if error rate moves.

Phase 4 — Expand by namespace tier

After tooling survives a week, I move to tier-2 services, then revenue-critical paths last. Revenue last is intentional: by then I trust the playbook and the allow-list templates, not my optimism.

I keep a shared library of policy fragments — dns-egress, ingress-from-openshift-router, scrape-from-monitoring — in Git, reviewed like any other platform code. Copy-paste with label typos is how prod breaks.

Zero trust without zero communication

Microsegmentation is a team sport. I ask service owners for three lists before I touch their namespace:

  1. Who must call you (ingress sources)
  2. Who you must call (egress destinations, ports)
  3. What breaks if DNS or NTP is slow (usually everything)

I also tell them what will not change without notice — monitoring paths, deploy pipelines, emergency break-glass.

Break-glass matters. I maintain a documented way to disable or bypass policies in an incident — temporary namespace annotation if your CNI supports it, Git revert path, or a known kubectl delete networkpolicy with audit logging. Security teams flinch at this. I explain that an undocumented bypass is what happens at 3 a.m. anyway, except someone does it in panic without telling anyone. Named break-glass is honesty.

Testing policies before production argues back

Staging must match production CNI behavior. Minikube default networking taught me lessons that did not transfer to OpenShift OVN. I test:

Happy path — user journey, health endpoints, background jobs

Deploy and rollback — new Pods get same labels; policies select labels, not Deployment names. A label change during rollout can orphan traffic.

Scale events — HPA adds Pods; policies attach by selector, usually fine, unless you relied on Pod IP rules that do not scale mentally

Failure injection — deny a non-critical egress on purpose in staging and confirm alerts fire

Cilium has policy verdict logging. Calico has audit rules. Use what your platform gives you. “We think this policy allows traffic” is weaker than “we saw allow verdicts in logs.”

Common mistakes I still watch for

Wrong podSelector or namespaceSelector labels. matchLabels must match exactly. Typos create deny-by-default for selected Pods instantly.

Forgetting ingress to readiness probes. Kubelet probes run on the node network path; generally not blocked by NetworkPolicy for HTTP/TCP probes to the Pod IP. But if you use exec probes only, different story. If something custom scrapes readiness from another Pod, you need a rule.

Only allowing ingress, not egress. App cannot call database; you fix ingress to app; database still unreachable because egress from app is denied.

IP blocks without maintenance plan. SaaS IP ranges change. CIDR rules rot. Prefer namespace selectors or egress gateways where possible.

Policies in Git with no owner. Orphan policies accumulate. Someone deletes the Service but leaves allow rules; someone adds a Service and never opens the port.

I keep a namespace diagram updated quarterly — even a Mermaid chart in the repo — because my memory is not a CMDB.

OpenShift notes from the field

OpenShift adds Router, Ingress Controller, and sometimes EgressRouter or EgressFirewall objects at project level. AdminNetworkPolicy can enforce baselines cluster-wide. I coordinate with cluster admins before namespace policies fight platform policies.

The router namespace label matters for ingress allow rules. Monitoring often lives in openshift-monitoring. DNS is still kube-system or dns-default depending on version. I read the cluster docs for the version I am on instead of trusting a three-year-old blog post — including mine, if you read this in 2029.

Sustainable zero trust

Perfect isolation on day one is a fantasy. Progressive enforcement with measured rollback is how production stays up while security improves.

I no longer argue that NetworkPolicy is optional for every cluster. I also no longer argue that default deny everywhere by Friday proves professionalism. The humble path is:

  • Map flows
  • Allow DNS and metrics first
  • Default deny one safe namespace
  • Add allows one rule at a time
  • Expand tier by tier
  • Document break-glass

Zero trust is not “trust nothing on Tuesday.” It is verify each flow, encode it explicitly, and stop when the error budget says stop. I have broken prod with a single YAML file. I have also prevented a compromised test Pod from reaching a production database because egress was denied by a policy we tested the month before. Both stories are true. The second one only happened because we went slow enough to survive the first.

If you are starting NetworkPolicy this week, pick one namespace, allow DNS egress, watch dashboards for an hour, and only then write the word deny in a manifest. Your on-call future self will thank you. Mine did.