Over the last week, I’ve encountered two incidents where an incorrect DNS entry was created and started causing problems. In one case, Varnish wouldn’t start because one of the backends resolved to multiple IPs. In the other, a Mongo node was pointing at the wrong IP. In both cases, the DNS entry was corrected and the changes propagated, but the problem hosts themselves were still seeing the stale entries.
I remembered that we were running nscd, so I restarted the service to flush the cache. Unfortunately, that didn’t help. A colleague informed me that nscd‘s cache is actually persistent by default on CentOS 6. To flush it, you need to run
nscd -i hosts
-i stands for invalidate.
You can also check if nscd‘s cache is persistent by running
and looking for
cache is persistent. Note that
nscd caches several other types of data (not just DNS entries), so if you’re looking for the DNS cache specifically, make sure to look under the
hosts cache section.
This tip was reposted from my blog, jontai.me