Plenty of blogs I want to follow don't publish an RSS or Atom feed. They have a perfectly good listing page — a grid of post cards, a list of headlines — but no machine-readable feed behind it. So I built the piece of SlashFeed that closes that gap: BoringRSS.ListingScraper, an extraction engine that turns a listing page into structured entries using nothing but CSS selectors.
This post is about the discovery layer — how we go from a URL to a set of {title, url, date, ...} entries. The headless browser client that renders JavaScript SPAs is its own separate concern (there's a dedicated doc for the Browserless sidecar); here I only care about how the listing page gets read.
Two stages that don't know about each other
The pipeline has exactly two stages, and keeping them independent was the most important design decision.
def run(url, rules, opts \\ []) do
with {:ok, html} <- fetch_html(url, opts) do
extract_entries(html, rules, url)
end
end
Stage one, fetch_html/2, acquires raw HTML. Stage two, extract_entries/3, parses it with Floki and applies selectors. That's the whole show. The reason they're split is testing and iteration: extraction can be exercised against a saved HTML snapshot with no network at all, and the fetch mechanism can change without touching a line of extraction logic.
fetch_html/2 takes a :mode:
def fetch_html(url, opts) do
mode = Keyword.get(opts, :mode, :http)
case mode do
:http -> fetch_html_http(url, opts)
:headless -> BoringRSS.HeadlessFetcher.fetch_page(url, opts)
other -> {:error, {:invalid_mode, other}}
end
end
:http is a plain Req.get/2 with a sane User-Agent and redirect following. It's the default and it's what works for the overwhelming majority of sites — anything server-side rendered (Next.js SSR, Hugo, Jekyll, static HTML). :headless delegates to a real Chromium instance for the sites where the DOM is empty until JS runs. The headless path is a last resort, not a default, because it's slow and needs a running sidecar. Most pages never need it.
One detail I'm happy with: the HTTP fetch can inject a Req.Test plug, resolved from options or app config, so tests stub the network cleanly instead of monkeypatching.
Rules are just a map of selectors
A rule set is a plain map with string keys. Only post_selector is required — it identifies the container element for each post. Everything else is optional and evaluated relative to each matched container:
%{
"post_selector" => "article.post-card",
"title_selector" => "h2",
"url_selector" => "a",
"date_selector" => "time",
"summary_selector" => "p.excerpt"
}
Extraction maps each container into an entry and then filters aggressively:
containers
|> Enum.map(fn container -> extract_one(container, rules) end)
|> Enum.reject(fn entry -> is_nil(entry.title) and is_nil(entry.url) end)
|> maybe_resolve_urls(base_url)
An entry with neither a title nor a URL is noise, so it's dropped. The per-field extractors are deliberately forgiving. extract_href/2 first looks for a matching child link, and if the selector matches nothing it falls back to checking whether the container itself is an <a> — a pattern that shows up constantly with card-as-link layouts. extract_date/2 prefers a machine-readable datetime attribute and only falls back to text. extract_src/2 tries src then data-src to survive lazy-loaded images. And every Floki call is wrapped so a malformed selector logs a warning and returns [] instead of crashing the whole run.
Relative URLs get resolved against the source URL via URI.merge/2, which is why extract_entries/3 (with a base URL) is the version you actually want.
Finding selectors without guessing
Writing selectors blind is miserable, so there's a small discovery toolkit. discover/1 throws a list of common blog/news container patterns at the parsed document and reports which ones match, sorted by hit count:
@common_post_selectors [
"article",
"div[class*='post']",
"div[class*='card']",
"div[class*='blog']",
...
]
It also prints counts for common title, link, and date sub-selectors, so you get a feel for the page's shape in one call. inspect_structure/3 then drills into a specific selector, showing the tag, attributes, a text preview, and child-tag frequencies for the first few matches. explore/2 chains the whole thing — fetch, discover, then auto-attempt an extraction using the best candidate — as a one-shot from iex.
For sites I've already figured out, examples/0 is a library of predefined configs with notes, keyed by domain. The Kaggle entry, for instance, documents that it's a true SPA that returns an empty <div id="root"> over plain HTTP and needs mode: :headless, and that comment links have to be filtered out with a[href*='/blog/'][target='_blank']. try_example/1 runs any of them end to end. These aren't magic — they're starting points you refine with inspect_structure/3.
Before anything gets saved, validate_rules/1 checks that post_selector is present, rejects unknown keys, and enforces that every value is a non-empty string. It returns a list of error tuples (empty means valid), which the admin UI surfaces live.
The admin test bench
All of this is wired into a LiveView at ListingScraperLive. You paste a URL and a JSON rules blob, toggle between HTTP and headless mode, and hit Fetch, Extract, or Fetch & Extract. Discovery results render as a table where each candidate has a "Use" button that drops it straight into the rules editor as your post_selector. Rules are validated on every keystroke.
Fetching runs under Task.Supervisor.async_nolink/2, not send(self(), ...), so a 30-second headless render never blocks the LiveView from handling other events. When the entries look right, "Save as SPA Feed" persists a feed with type spa_listing, storing the rules and fetch mode. From there it's no longer the discovery layer's problem: a separate SpaFeedWorker and ContentScrapeWorker handle scheduled listing refreshes and full-article fetches. The ListingScraper only ever answers one question — what's on this page right now — and it answers it the same way whether you're poking at it in iex or clicking around the backoffice.