A public feed index for discovery and SEO

· · 30 views
Table of Contents

Most of SlashFeed lives behind a login. Your feeds, your reading stats, your categories — all private, all scoped to your account. But a private-by-default reader has a discovery problem: nobody outside the app can see that it exists, and search engines have nothing to index except a marketing page. So I built a public, unauthenticated feed directory at /index. This post is about what it surfaces, how it stays private-safe, and why it turned out to be the most important SEO surface we have.

Who it's for

The audience is people who are not logged in. A visitor lands on /index, sees a hero with live counters, and can do three things without an account: search the catalog, browse feeds by topic, and submit a new feed URL. The LiveView mounts with no session requirement — the only thing it grabs from the connection is the peer IP, for rate limiting:

def mount(_params, _session, socket) do
  remote_ip =
    case Phoenix.LiveView.get_connect_info(socket, :peer_data) do
      %{address: address} -> address |> :inet.ntoa() |> to_string()
      _ -> nil
    end
  ...

That IP is the only per-visitor state. There's no user, no tenant, no auth pipeline. It's a genuinely public page.

What gets surfaced — and what doesn't

This is the part I care most about getting right. SlashFeed is multi-tenant; the feeds table holds everyone's feeds, including private ones an individual user added to their own account. The public index must never leak those. The rule is one line, and it appears in every query that backs the page: where: is_nil(f.owner_id).

A feed with an owner_id belongs to a specific user — private, never public. A feed with a nil owner is a catalog feed: it was added through the public submit box or seeded into the shared directory, and belongs to nobody. That's the opt-in model. There is no "make this public" toggle a user can accidentally flip. Owned feeds are private, full stop; only ownerless catalog feeds are ever listed. Search enforces the same thing:

def search_public_feeds(query, limit) do
  like = "%#{query}%"

  from(f in Feed,
    where: ilike(f.title, ^like) or ilike(f.url, ^like) or ilike(f.description, ^like),
    where: f.feed_type == "rss_atom" or is_nil(f.feed_type),
    where: is_nil(f.owner_id),
    where: not like(f.url, "mailto:%"),
    order_by: [desc_nulls_last: f.last_fetched_at],
    limit: ^limit
  )
  |> Repo.all()
end

The topic listing (public_index_topics/1, list_public_feeds_by_topic/2) carries the exact same three filters: RSS/Atom only, owner_id nil, and no mailto: URLs (newsletter inboxes are per-user by nature, so they're excluded too). I deliberately keep that query in the Feeds context rather than inline in the LiveView, so there's a single tested place where the privacy predicate lives. The LiveView just delegates:

defp search_feeds(q), do: Feeds.search_public_feeds(q)

If I ever add a fourth privacy condition, it goes in one module and every entry point inherits it. That's the whole reason the search isn't written inline in handle_event/3 even though it's a one-liner.

The SEO payoff

Here's the thing I didn't fully appreciate when I started: a public directory of real feeds is enormously better SEO than a marketing page. Every catalog feed has a title, a description, and a domain — that's real, crawlable content that maps to what people actually search for. The landing page can say "RSS reader" a hundred times; the index page can rank for the actual feeds people are looking for.

To make that work, the page had to be crawler-friendly, not a JavaScript-only shell. Two decisions matter. First, /index accepts a ?q= query param and renders results server-side on the initial load, so a crawler (or a Google sitelinks search box) hitting /index?q=elixir gets real HTML back:

# Support `/index?q=…` so the page is a valid Google `SearchAction` target
# (the sitelinks search box): results are rendered server-side on load.
def handle_params(params, _uri, socket) do
  case params |> Map.get("q", "") |> to_string() |> String.trim() do
    "" -> {:noreply, socket}
    q ->
      {:noreply,
       socket
       |> assign(:search, q)
       |> assign(:searching, byte_size(q) >= 2)
       |> assign(:search_results, search_feeds(q))}
  end
end

Second, the landing page's structured data points its SearchAction here. In BoringRSSWeb.SEO.home_json_ld/0, the WebSite node declares the index as its search target:

"potentialAction" => %{
  "@type" => "SearchAction",
  "target" => %{
    "@type" => "EntryPoint",
    "urlTemplate" => "#{base}/index?q={search_term_string}"
  },
  "query-input" => "required name=search_term_string"
}

The comment in that module spells out the tradeoff: our authenticated /search route is blocked in robots.txt, so it can't back a sitelinks search box. /index can, because it's public and server-rendered. The SEO module builds every bit of JSON-LD as an Elixir map encoded with Jason.encode!(map, escape: :html_safe), and the layout renders whatever's in the json_ld assign into a <script type="application/ld+json"> block in <head>. The index page leans on the standard meta tags via page_title:

|> assign(:page_title, "Feed Index — SlashFeed")

I could go further and emit an ItemList of catalog feeds as JSON-LD on the index itself. I haven't yet — the SearchAction + server-rendered results already do the heavy lifting, and I'd rather ship the ranking win before over-engineering the structured data.

Relationship to the landing page

Think of the landing page and /index as a funnel with two jobs split cleanly. The landing page sells the product: it carries the Organization, SoftwareApplication, and pricing Offer nodes in its JSON-LD. The index page is the product's open face — it proves SlashFeed reads real feeds, gives crawlers something worth indexing, and lets a curious visitor submit a feed (rate-limited to five per IP per hour via Hammer) before ever creating an account. One page tells search engines who we are; the other gives them a reason to keep coming back.

The nice property is that they reinforce each other through structured data: the landing page's WebSite node hands its search box to /index, and /index fills that box with genuine, owner-less catalog content. No leaked private feeds, no JavaScript-only dead ends — just the open web, indexed.