elixir oban headless-browser scraping browserless

Fetching JS-rendered feeds with a headless browser

· · 32 views
Table of Contents

Most of SlashFeed is a boring, happy RSS/Atom pipeline: fetch XML over HTTP, publish it to RabbitMQ, parse it, upsert entries. That covers the overwhelming majority of feeds. But a growing slice of the sites people actually want to follow don't publish a feed at all — and worse, a chunk of those render their content entirely in the browser. The listing page you get from a plain GET is an empty <div id="root"> and a bundle of JavaScript. Kaggle's blog is the canonical example.

This is the story of SpaFeedWorker: how we fetch and pseudo-parse feeds from JS-rendered single-page-application sites, and the tradeoffs that pushed it onto its own queue with its own rate limiting.

Why the normal pipeline can't help

The standard path is FeedFetchWorker. It does exactly one thing well: HTTP GET a feed URL and hand the bytes off to be parsed. It's careful about the things that matter for RSS — conditional requests via ETag/Last-Modified, manual redirect following so it can tell a permanent 301 from a temporary 302 and rewrite feed.url accordingly. What it explicitly refuses to do is guess at HTML.

In fact it bails out the moment it sees a non-standard feed:

cond do
  feed.feed_type == "spa_listing" ->
    Logger.info("[#{__MODULE__}] Skipping spa_listing feed #{feed_id} — use SpaFeedWorker")
    {:cancel, "spa_listing feeds are handled by SpaFeedWorker"}

That {:cancel, ...} is a guard rail for stale jobs — the routing decision actually happens upstream, in the scheduler.

Routing: how a feed ends up on the JS queue

FeedSchedulerWorker runs on Oban Cron every five minutes, picks the feeds that are due, and decides which worker each one belongs to. The whole routing rule is two function clauses:

defp worker_for_feed(%{feed_type: "spa_listing"} = feed) do
  {SpaFeedWorker, %{feed_id: feed.id}}
end

defp worker_for_feed(feed) do
  {FeedFetchWorker, %{feed_id: feed.id}}
end

So the discriminator is a single column: feed_type. If it's "spa_listing", the job goes to SpaFeedWorker on the :js_fetch queue; everything else takes the normal XML/Atom pipeline. There's no probing or content-sniffing at fetch time — a feed is classified once (when it's added) and the scheduler just honours that.

What SpaFeedWorker actually does

SpaFeedWorker runs on a dedicated queue with deliberately low concurrency:

use Oban.Worker,
  queue: :js_fetch,
  max_attempts: 3,
  unique: [fields: [:args], period: 300]

The :js_fetch queue exists precisely because headless rendering is heavy — roughly 200–400 MB of RAM per Chromium session. You do not want that competing for slots with hundreds of cheap XML fetches, so it's isolated and capped (default concurrency 2). The unique window also stops the same feed being rendered twice inside five minutes if a manual refresh races the scheduler.

On perform/1 the worker re-validates its assumptions before doing anything expensive: the feed must have spa_listing_rules configured and must still be feed_type == "spa_listing", otherwise it cancels. Then it resolves a fetch mode and delegates to ListingScraper:

defp resolve_fetch_mode(feed) do
  if feed.js_rendered && BoringRSS.HeadlessFetcher.enabled?() do
    :headless
  else
    :http
  end
end

This is the important design decision: not every spa_listing feed needs a browser. Plenty of sites lack an RSS feed but still server-side render their content (Next.js SSR, Hugo, Jekyll, static HTML). For those, a plain HTTP GET gives you a full DOM and headless Chrome would be pure waste. So we only reach for the browser when the feed is explicitly flagged js_rendered: true and the sidecar is switched on in config. Everything else falls back to :http.

Either way the extraction step is identical — ListingScraper.run/3 fetches the HTML (via HeadlessFetcher or Req, depending on mode), parses it with Floki, and applies the feed's spa_listing_rules CSS selectors (post_selector, title_selector, url_selector, and friends) to pull out entries. I like that the fetch mechanism and the extraction logic are cleanly separated: the selector rules can be developed and tested against saved HTML snapshots in iex with no browser in the loop at all.

The headless fetcher

HeadlessFetcher talks to a Browserless-compatible Chromium container (ghcr.io/browserless/chromium) over plain HTTP. It's intentionally stateless — it renders a URL and returns the fully-rendered DOM as an HTML string. Two details earned their keep in production.

First, we encode the request JSON by hand rather than using Req's json: helper:

# Req's `json:` also sets `accept: application/json`, which causes
# Browserless v2 to 404 — it uses content negotiation and won't serve
# rendered HTML when the client asks for JSON responses.
defp post_render(full_url, body, recv_timeout) do
  encoded = Jason.encode_to_iodata!(body)

Second, different Browserless versions expose the render endpoint at different paths (/chromium/content, /content, /chrome/content). If the configured path 404s, the fetcher walks the fallbacks and caches whichever works in a :persistent_term for the life of the node, so subsequent calls skip the probing. There's also a probe/0 helper that prints a little table of which paths respond — genuinely useful when a sidecar upgrade silently moves the endpoint.

Rate limiting and quota

Rendering is expensive on both ends — our RAM and, if you're on hosted Browserless, your session quota. So SpaFeedWorker throttles per-domain before it even calls the scraper:

defp check_rate_limit(url) do
  domain = url |> URI.parse() |> Map.get(:host, "unknown") |> strip_www()

  case Hammer.check_rate("spa_fetch:#{domain}", 30_000, 1) do
    {:allow, _count} -> :ok
    {:deny, _limit} -> {:error, :rate_limited}
  end
end

One render per domain per 30 seconds. When it trips, the job errors with a "rate limited — will retry" message and Oban's backoff handles the retry — no busy-looping against the sidecar. Combined with the low :js_fetch concurrency, this keeps a burst of SPA feeds from stampeding Chromium.

Downstream: enrichment

The entries SpaFeedWorker produces are deliberately thin — often just a title, URL, and whatever summary text the listing card exposed. So the content field starts life as the summary, with a comment flagging that it "may be enriched later by ContentScrapeWorker." If the feed has scraper_enabled, that second worker fetches each article page and runs the Readability/CSS extraction pipeline — and it has the same js_rendered escape hatch, rendering the article page through HeadlessFetcher when needed. Same tool, two stages: the browser gets you the listing, then optionally the article body.

What I'd flag

This subsystem is the least "boring" part of an intentionally boring reader, and that's the point — it's fenced off. A separate queue, a hard per-domain rate limit, an off-by-default sidecar, and a fast :http path for anything that doesn't truly need a browser. The failure modes (:all_paths_failed, :rate_limited, sidecar :disabled) are all handled without taking down a fetch cycle. If Browserless is down, enabled?() short-circuits and those feeds simply skip a beat rather than erroring loudly. For a feature that exists to scrape sites that never wanted to be scraped, that restraint is the whole design.

All posts RSS Feed