Fixing spurious :nxdomain errors in Elixir with dnsmasq and :inet_res

For a few weeks SlashFeed kept accumulating the same fetch error on feeds that were, when I looked, perfectly healthy:

Request failed: :nxdomain

It came in bursts — sometimes more than a hundred feeds at once — and it hit unrelated domains: ziglang.org, oban.pro, slack.engineering, dozens more with nothing in common except that I fetched them in the same minute. Each one then went onto the app's own exponential backoff, which is hours, for a problem that had already vanished by the time I looked.

This post is the debugging story in the order it actually happened, because the interesting part is how many plausible fixes only moved the number a little — and what finally explained it.

:nxdomain does not mean the domain doesn't exist

The first thing to unlearn. On the BEAM, :nxdomain is not "this name has no records". It is the reason Erlang's resolver reports for a lookup that failed for any reason — including a dropped or timed-out UDP packet. A real NXDOMAIN and a lost datagram look identical to your code. If you are branching on :nxdomain to decide that a host is dead, you are branching on a lie.

That reframed the problem. These were not dead domains. Something between the VM and the nameserver was intermittently failing, and the question was where.

Suspect one: the upstream forwarder, under concurrency

The setup is unglamorous: the machine's DNS was a small shared forwarder, which in turn passes every query on to whatever upstream it is configured with. The :feed_fetch Oban queue ran 25 jobs at once, and every job resolves a different host — so one poll cycle fires a burst of simultaneous lookups through one cheap forwarder.

I checked it directly rather than assuming. Firing 25–40 concurrent dig lookups at real feed hostnames reproduced real failures: 6 of 30 hostnames failed in a single burst, and querying the same six again individually, seconds later, resolved five of them immediately. Genuine, transient, concurrency-correlated blips. Not dead domains, not a parser bug.

What I tried, in order

#	Change	Effect
1	Lowered :feed_fetch concurrency, 25 → 10 → 5	Fewer lookups per burst
2	In-process retry for transient transport errors	Absorbs a blip that clears within ~10 s
3	Pointed the machine straight at public resolvers (1.1.1.1 / 8.8.8.8)	Removed the forwarder, but a burst against 1.1.1.1 could still occasionally fail
4	A local caching resolver, dnsmasq	125 feeds in :nxdomain dropped to 8

The retry is worth a few lines, because the delay is the part I got wrong first:

defp fetch_with_retry(url, opts, attempts_left) do
  case Req.get(url, opts) do
    {:error, %{reason: reason}}
    when attempts_left > 0 and reason in @transient_transport_reasons ->
      Process.sleep(transport_retry_backoff_ms())
      fetch_with_retry(url, opts, attempts_left - 1)

    result ->
      result
  end
end

It retries only the transient transport reasons (:nxdomain, :timeout, :econnrefused, :closed, :ehostunreach, :enetunreach) — never an HTTP status, and never something permanent like a certificate error, which would fail identically every time. I first tried a 250 ms delay and it did not reliably help. Under a burst, resolving needed seconds to recover once queried in isolation, so the default is 5 s, two extra attempts. The reason it matters: the next Oban attempt is scheduled with exponential backoff measured in hours, so retrying in-process first is what turns a blip into a success instead of a parked feed.

dnsmasq was the step that moved the number. It coalesces duplicate concurrent queries into one upstream request and caches the answers, so most of a burst never leaves the machine. Its configuration is tiny, but two lines matter:

listen-address=127.0.0.1
bind-interfaces
cache-size=10000

Loopback only, so it can never become an open resolver reachable from the network. And cache-size — the default is 150 entries, which is far too small for a process that resolves hundreds of distinct feed hosts.

I considered it done. It was not.

The failures that survived

Even with dnsmasq in place, roughly 1 000 :nxdomain discards a day continued, spread over about 600 distinct hosts. Three facts made that strange:

• 99 % of those feeds had also fetched fine in the previous two days, so they were not dead.
• It was not concurrency: failures were the same at 1 and at 60 concurrent lookups.
• They came back in milliseconds, not after a timeout. A dropped packet takes a while to fail. This failed instantly.

That last one is the clue I should have read sooner. An instant failure is not a lost datagram; it is a received error.

The AAAA record

So I stopped theorising and measured on the running node, sampling 120 hosts from the recent failures and resolving them three rounds in a row, through different resolvers:

Resolver	Failed per round (120 hosts)
BEAM default (lookup: [:native] → macOS getaddrinfo)	35, 36, 35
BEAM :inet_res (Erlang's own client), same dnsmasq	1, 1, 0
dig for the A record against dnsmasq / 1.1.1.1 / 8.8.8.8	fine

Same machine, same dnsmasq, same hosts. The only difference was which resolver asked.

The difference is the AAAA record. macOS getaddrinfo looks up the A and the AAAA record together and fails the entire call if either sub-query errors; Erlang then reports any failure as :nxdomain. Of the 41 hosts the native resolver failed on, 31 returned SERVFAIL for AAAA through dnsmasq while the A record was a perfectly good NOERROR. The public resolvers, queried directly, answered those same AAAA queries correctly — so dnsmasq, or its hop upstream, was intermittently answering SERVFAIL for a record type this network cannot even use, because it has no IPv6 route.

The A record was fine the entire time. The fetch died on a record I could never have connected to anyway.

The fix: use Erlang's resolver

By default the BEAM delegates name resolution to the OS (lookup: [:native]). The fix is to make it use its own DNS client instead, which asks only for the A record on an IPv4 socket, retries, and walks the nameserver list on failure:

def configure(opts \\ Application.get_env(:boring_rss, __MODULE__, [])) do
  if Keyword.get(opts, :enabled, true) do
    :ok = :inet_db.set_lookup([:file, :dns])
    add_fallbacks(Keyword.get(opts, :fallback_nameservers, []))
  end

  :ok
end

defp add_fallbacks(fallbacks) do
  known = nameservers()

  for ns <- fallbacks, ns not in known do
    :inet_db.add_ns(ns)
  end

  :ok
end

It is called first thing in Application.start/2, before anything resolves. lookup: [:file, :dns] keeps /etc/hosts and swaps the OS resolver for :inet_res; the fallback list (1.1.1.1, 8.8.8.8) is appended after the system's dnsmasq, so when the local resolver misbehaves or is down, the lookup walks on to the next nameserver instead of failing. The result on the same 120-host sample: 35 failures per round became 0–1.

Two properties are worth stating plainly. A real NXDOMAIN is unchanged, so genuinely dead domains still fail and back off exactly as before — I removed a failure mode, not the ability to detect dead hosts. And it can be applied to a running node without a restart and reverted just as easily:

:inet_db.set_lookup([:file, :dns])
:inet_db.set_lookup([:native])   # roll back

What it does not cover: mDNS / .local names, which the native resolver handled and nothing in the app uses, and dnsmasq's own intermittent SERVFAIL, which is now absorbed by the retry and the fallbacks rather than fixed.

What will never reach zero

After all of this there were still a handful of feeds in :nxdomain. I checked the last eight against two independent public resolvers at the same moment:

• Four resolved fine on both — already self-healed, gone on the next scheduled fetch.
• One failed on one resolver and succeeded on the other — provider-specific flakiness.
• Three failed on both — the domain's own DNS was genuinely broken at that moment, independent of my machine, my network and my resolver setup.

That last group is not something a local fix can solve, and it should not be. The correct behaviour is the one the app already had: record the error, back off, retry later, and clear it on the first success.

Takeaways

• On the BEAM, :nxdomain means "the lookup failed", not "the name does not exist". Do not use it to decide a host is dead.
• Measure before you theorise. Concurrent dig against real hostnames, then the same hosts through each resolver on the running node, turned guesses into numbers at every step.
• Read the timing. A failure that returns in milliseconds is a received error, not a lost packet — and that single observation was the key.
• getaddrinfo resolves A and AAAA as one all-or-nothing call. On a network with no IPv6, an upstream that is flaky about AAAA can fail lookups for records you will never use.
• A local caching resolver (dnsmasq) is cheap and effective for a process that resolves hundreds of hosts: coalesced queries, a real cache size, bound to loopback only.
• Retry the transient transport errors in-process, with a delay sized to the failure, before handing the job back to a backoff measured in hours.
• Use :inet_db.set_lookup([:file, :dns]) plus fallback nameservers, and keep it reversible. It removed 35 failures out of 120 without losing the signal from genuinely dead domains.

None of the first three fixes was wrong; each one removed a real cause of failures, and the numbers went down every time. The lesson is that "it got a lot better" is the moment to keep measuring, because the remaining thousand a day were a different bug entirely.