SlashFeed already had a circuit breaker for feed fetching. When a feed answers with a 403, 429 or 503, the worker classifies the refusal, records it, and parks that one feed on a backoff that grows from an hour to a week. It works, and it is the right shape for a single misbehaving feed.
It is the wrong shape for a publisher. Medium, for example, serves hundreds of feeds from one domain — one per author — and it rate-limits by IP, not by feed. When it starts answering 429, every one of those feeds is refused, and each one has to be refused on its own before it learns to back off. The per-feed breaker is learning the same lesson hundreds of times, and every lesson is another request to a server that has already told us to stop.
So I added a second breaker, one level up: per domain. This post is how it works — the counting, the state, the gate in the worker, a request budget that lives on the same row, and the admin page on top. Along the way there are two Ecto gotchas and one decision I would make differently next time.
What it does
The rule is deliberately simple. If a domain answers more than 2 times with a 429 or a 5xx inside 60 seconds, the breaker opens for one hour. While it is open, FeedFetchWorker does not make the request at all. All three numbers are configurable, globally and per domain.
Three details turn that sentence into code:
- What counts as an error. A
429or any5xx. Not a403— those are the per-feed breaker's business, and a404is just a feed that moved. The breaker is about a server that is struggling or refusing us, not about a bad URL. - What a domain is.
maximebeauchemin.medium.comandmedium.commust share a breaker, because they share an IP and a rate limit. - What "open" does to a job. It must not count as a failure of the feed, and it must not leave the feed scheduled for a moment inside the open window, where it would just be skipped again.
Finding the domain
The right answer is the public-suffix list, and there is no public-suffix library in the dependency tree. I did not want to add one for this, so domain_key/1 keeps the last two labels of the host — and three when the shape is the common *.co.uk one: a two-letter TLD preceded by co, com, org, net, gov, edu or ac.
defp take_registrable(labels) do
[tld, second | _] = Enum.reverse(labels)
if String.length(tld) == 2 and second in @second_level and length(labels) >= 3 do
Enum.take(labels, -3)
else
Enum.take(labels, -2)
end
end
IP literals and single-label hosts are returned whole. It is wrong for exotic suffixes like *.pvt.k12.ma.us, and I decided that is fine: the cost of a wrong guess is a breaker that is slightly too wide or too narrow, and an admin can close it with one click. A heuristic you can see and override beats a dependency you have to trust.
Counting in Hammer, deciding in Postgres
The breaker needs two kinds of memory, and they have nothing in common.
The error rate is a counter over a sliding-ish window. That is exactly what Hammer is for, and SlashFeed already runs Hammer with a Redis backend whenever REDIS_URL is set, so counts are shared across nodes for free. One bucket per domain:
case Hammer.check_rate("domain_cb:" <> domain, window * 1_000, threshold) do
{:deny, _limit} -> {:tripped, trip(domain, status, "auto")}
_ -> :ok
end
check_rate/3 allows threshold calls per window and denies the next one. "More than 2 errors in 60 seconds" is therefore exactly "the call Hammer denies" — the third error trips the breaker, with no counting logic of my own.
The breaker state is the opposite: it has to be listed, edited by an admin, kept across a restart, and survive someone flushing Redis. That is a table, domain_circuit_breakers, with one row per domain that has ever tripped (rows are created lazily, so the table is as small as the number of misbehaving domains). And the state is a single column:
def open?(%Breaker{opened_until: nil}, _now), do: false
def open?(%Breaker{opened_until: until}, now), do: DateTime.compare(until, now) == :gt
There is no status field and no sweeper. A breaker is open if opened_until is in the future, full stop; when the hour passes it closes itself, because nothing needs to do anything for time to pass. It is the same trick as an expiring session token, and it removes a whole category of bug — the breaker that stays open forever because the job that was meant to close it died.
Opening it exactly once
Tripping is a write that many processes can attempt at the same moment: the queue runs jobs concurrently, and when a domain starts failing, several in-flight requests fail together. Two things must not happen. A late error must not extend an open breaker, and it must not inflate the trip count. The upsert encodes both:
on_conflict =
from b in Breaker,
update: [
set: [opened_until: ^until, opened_at: ^now, opened_by: ^opened_by,
last_status: ^status, updated_at: ^now],
inc: [trip_count: 1]
]
on_conflict =
if opened_by == "auto",
do: where(on_conflict, [b], is_nil(b.opened_until) or b.opened_until <= ^now),
else: on_conflict
An automatic trip only wins when the row is closed or expired; a manual one (from the admin page) always restarts the clock. Errors that arrive while the breaker is already open are not even counted in Hammer — count_error/2 checks the row first and returns early.
The gate in the worker
The worker asks one question before doing anything that touches the network:
case DomainCircuitBreaker.admit(feed.url) do
{:throttled, wait_ms} -> {:snooze, div(wait_ms, 1_000) + 1 + :rand.uniform(5)}
{:open, until} ->
defer_until_circuit_closes(feed, until)
{:cancel, "domain circuit breaker open until #{until}"}
:ok -> do_fetch_rss_atom(feed, job)
end
An open circuit returns {:cancel, …}: no request, no fetch_error, no failure count, no ticket. The feed did nothing wrong. But the scheduler had already claimed the job and pushed next_fetch_at forward by the feed's own interval, which could be a day away — or land inside the open window and be skipped again. So the worker sets next_fetch_at to the moment the breaker closes (or to the feed's own per-feed blocked_until if that is later, because that is a different breaker with its own opinion).
The other half, recording the response, has one ordering subtlety. SlashFeed retries a 429 once through Tor, because a different exit IP often gets past a per-IP limit. If the breaker looked at the final result, a successful Tor retry would hide the fact that the domain is rate-limiting us. So the direct response is recorded before the Tor retry:
feed.url
|> follow_redirects(opts, false, @max_redirect_hops)
|> record_domain_outcome(feed)
|> maybe_retry_via_tor(feed, opts)
|> handle_response(feed, job)
A request budget on the same row
Once the row exists, a second knob is nearly free. The breaker answers "stop, this domain is unhappy"; sometimes what I want is "go gently, this domain is fine but I do not want to send it 90 requests a minute". Each breaker row can carry a requests_per_10s, null meaning unlimited.
The unit is deliberate. I started with requests per second, and the rate I actually wanted for a polite crawl was something like three every ten seconds — which, per second, is 0.3, a number nobody wants to type or read back. A whole number per ten seconds is something you can say out loud.
It reuses Hammer again, turning the figure into spacing:
def rate_window_ms(per_10s) when is_integer(per_10s) and per_10s > 0,
do: max(div(10_000, per_10s), 1)
Ten per ten seconds is one request a second; three is one every 3.3 seconds. I enforce it as evenly spaced requests — a bucket of one per 10 000 / N ms — rather than N at once at the start of each ten-second window, because the whole point is not to trip the publisher's own limiter, and a burst is exactly what trips it. When the budget is spent, admit/1 returns {:throttled, window_ms} and the worker snoozes the job past the window, with a few seconds of jitter so the jobs that were queued together do not all wake in the same instant. A throttled request is not an error: it does not count towards opening the circuit, and an open circuit always wins over the limit because it already forbids everything.
Changing the unit after the column had shipped was its own small lesson. The old requests_per_second had already run in production with real values on two domains, so the change is a new migration that adds requests_per_10s and converts the existing rows (round(rps × 10), at least one) — a row limited to one per second became ten per ten seconds, which is the same pacing it had before. The old column is left in place, unused, rather than dropped: nothing is lost, and the previous code can still be rolled back.
I want to be straightforward about the weakness. Hammer's windows are fixed, not sliding, so two requests can land close together across a window boundary. For politeness towards a publisher that is entirely acceptable. For a hard quota it would not be, and I would reach for a token bucket.
It fails open
A breaker is an optimisation — a feature that protects other people's servers and our own queue. It must never become the reason a working feed stops being fetched. Every public entry point rescues infrastructure errors, logs a warning, and answers as if the breaker were not there:
rescue
error ->
Logger.warning("[DomainCircuitBreaker] admit failed open: #{Exception.message(error)}")
:ok
catch
:exit, reason -> ...
This earned its keep immediately. The code shipped before the migration had run, and for a few minutes the log filled with relation "domain_circuit_breakers" does not exist while every feed kept fetching exactly as before. A breaker that fails closed would have turned a missing table into an outage.
The admin page
/admin/circuit-breakers lists every breaker with its state and time left, who opened it (auto or manual), the last status, the trip count and the effective parameters, with a * marking an override of the default. An admin can open or close a breaker, edit the three parameters plus the request rate, remove a row to return to the defaults, or add a domain before it ever trips. Closing a breaker also deletes its Hammer buckets — otherwise the very errors that opened it would still be in the window and re-open it on the next response.
The feeds page got a Circuit open card and filter, which was the more interesting query. The breaker is keyed by domain; feeds only have a URL. Rather than store a derived column, the query reads the host out of the URL in SQL and matches it against the short list of currently open domains, exactly or as a subdomain:
host_re = "^[a-zA-Z][a-zA-Z0-9+.-]*://(?:[^/@]*@)?([^/:?#]+)"
from(f in query,
where:
fragment("lower(substring(? from ?)) = ANY(?)", f.url, ^host_re, ^domains) or
fragment("lower(substring(? from ?)) LIKE ANY(?)", f.url, ^host_re, ^suffixes)
)
The regular expression is passed as a parameter on purpose. My first version inlined it in the fragment string, and Ecto counts every ? in a fragment as a placeholder — including the ? in (?:…) and in [^/:?#]. It refused to compile, complaining it expected ten arguments. Moving the pattern into a bound value sidesteps the whole thing. When no breaker is open the function returns where: false and the query never touches a row.
Testing something built on global counters
Hammer's counters are node-global and wall-clock based, which is exactly what you do not want in a test suite that runs hundreds of tests against one process. The worker tests all fetch from example.com and answer 429 and 500 on purpose; one file's errors would open the breaker for the next. So the breaker is switched off in config/test.exs, and the tests that are about it turn it on and use a domain that is unique to the test — "cb-#{System.unique_integer([:positive])}.test" — so they never share a bucket.
The rate-limit tests have one more trick. A fixed window makes "the second call is over budget" flaky if the first call lands just before a boundary. Using the slowest budget there is — one request per ten seconds, a ten-second window — makes that boundary a vanishingly small target, so the assertion holds wherever in the window the test happens to run.
What I would do differently
- No half-open state. When the hour passes the breaker is simply closed, and the next window has to exceed the threshold again before it re-opens — which costs a domain that is still hostile a few more refused requests. A proper half-open breaker lets one probe through and decides from it. I skipped it because the cost is a handful of requests an hour, but it is the first thing I would add.
- Fixed windows. Discussed above; fine for politeness, wrong for quotas.
- The domain heuristic. It will be wrong for some suffixes. If it ever matters, the public-suffix list is a dependency away.
- Counts are only as shared as Hammer. With Redis configured the error counts are cluster-wide; without it they are per node, and the effective threshold is multiplied by the number of nodes. The state itself, in Postgres, is always shared.
Takeaways
- A per-feed breaker cannot see a problem that lives at the domain. If the resource being protected is shared, the breaker must be keyed by what is shared.
- Split the two kinds of memory: let a counter library hold the rate (Hammer here), and put the decision in a table you can list, edit and audit.
- Make "open" a timestamp, not a status.
opened_untilcloses itself and removes the sweeper, and with it the failure mode where nothing ever closes the breaker. - Make the tripping write idempotent under concurrency: only a closed breaker can be tripped automatically, and late errors must neither extend it nor be counted.
- Decide what a skipped job means. Cancelling is not failing; reschedule the feed for the moment the circuit closes, and record the error before any retry that could hide it.
- A breaker must fail open. If the thing that protects your fetches can take them down, it is not a safety feature.
- Put the knobs on the same row, and the admin page that edits them becomes the whole operational story: tune a domain, open it by hand, close it when the publisher says it is fine.
The best part of the design is how little of it is clever. A counter that already existed, one timestamp column, one case in the worker. Everything else is deciding what should happen when a server tells you to stop — and then making that the easy path.