Kubernetes is a breeze they said. Now I‘m sitting here for several hours trying to find out why my pods randomly fail to resolve domain names.

Coming along my adventure: broken systemd configs, systemd-resolved stub causing loops, broken k3s modules and finding out that busybox‘s nslookup is broken for versions greater than v1.28.4.

50 issues later, I figured out that the dude who setup the corporate network (where the machine in question is located) uses two nameservers: one to resolve the internal routes and one for all the external domains. Luckily, coredns randomly picks a nameserver for each request. Therefore, sometimes queries for external domains reach the nameserver dedicated to the internal network which then answers with NXDOMAIN.

I hate networking so so much...