elixir cassandra xandra databases architecture

When to reach for Cassandra alongside Postgres

· · 33 views
Table of Contents

SlashFeed already had two databases before Cassandra showed up, and I want to be honest up front: adding a third datastore to a side project is usually a mistake. Every store is another thing to run, monitor, back up, and reason about. So the interesting question is not "is Cassandra cool" (it is) but "what class of data is bad enough in Postgres that a whole new operational surface pays for itself?"

This post walks through where that line landed for us, and — just as important — how I wired the client so Cassandra can sit in the tree switched off until there's a real write path that needs it.

Three stores, three jobs

The mental model I keep is one sentence per store:

  • PostgreSQL is the source of truth. Users, feeds, entries, subscriptions, OAuth grants — anything relational, anything I need transactions and foreign keys for, anything I'd cry about losing. It's the boring, correct default.
  • ClickHouse is for analytics. Aggregations over millions of read events, "how many articles were opened this week" style questions. Columnar, append-mostly, queried in fat batches.
  • Cassandra is for high-write-throughput, per-partition time-series that I want to expire on its own. The motivating case is per-user feed-entry read history: a firehose of tiny "user X read entry Y at time T" writes, always read back as "the last N reads for one user," and only worth keeping for a rolling window.

That last shape is genuinely awkward in Postgres. A read-history table is write-heavy, index-heavy, and grows without bound unless you bolt on a retention job. Cassandra's data model fits it almost embarrassingly well. Here's the table BoringRSS.Cassandra.EntryReads is built around:

CREATE TABLE IF NOT EXISTS entry_reads (
    user_id     uuid,
    read_at     timestamp,
    entry_id    uuid,
    feed_id     uuid,
    category_id uuid,
    PRIMARY KEY ((user_id), read_at, entry_id)
) WITH CLUSTERING ORDER BY (read_at DESC, entry_id ASC)
  AND default_time_to_live = 7776000;

The partition key is user_id, so every read for one user lives in a single partition — "last 20 reads" is one sequential scan, no fan-out across nodes. The clustering order is read_at DESC, so newest-first comes back with zero sorting. And default_time_to_live means Cassandra tombstones rows after 90 days by itself — no cron, no DELETE ... WHERE read_at < job, no vacuum pressure. The retention policy is a column definition. That's the pitch.

Setting it up: Docker, then one CQL statement

Cassandra runs as a cassandra:5 service in docker-compose.yml. The one gotcha worth internalising is that a fresh node takes 30–60 seconds to finish bootstrapping before it accepts CQL, so just cassandra-setup waits for nodetool status before doing anything:

docker compose exec -T cassandra nodetool status > /dev/null 2>&1 || \
    (sleep 10 && docker compose exec -T cassandra nodetool status > /dev/null)

Cassandra does not create keyspaces for you the way Postgres hands you a default database, so the setup task bootstraps the keyspace explicitly:

CREATE KEYSPACE IF NOT EXISTS boring_rss
  WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1}
  AND durable_writes = true;

SimpleStrategy with replication_factor: 1 is a single-node dev setting — in production you'd swap in NetworkTopologyStrategy and a real factor. Table DDL then lives in priv/cassandra/migrations/, applied in order by just cassandra-migrate.

The client: a thin, honest wrapper over Xandra

BoringRSS.Cassandra is deliberately small. All queries go through a single named cluster started in the supervision tree:

@cluster BoringRSS.Cassandra.Cluster

Xandra.Cluster maintains the pool of native-protocol CQL connections and reconnects automatically when a node drops. The child spec passes the keyspace down so every connection runs in the right context without a per-query USE:

def child_spec do
  {
    Xandra.Cluster,
    name: @cluster, nodes: nodes(), pool_size: pool_size(), keyspace: keyspace()
  }
end

The public API is four verbs. execute/3 runs a statement and returns {:ok, result} / {:error, reason}; prepare/2 prepares a statement; run/1 checks out a single connection so you can issue several statements on the same one; and each has a ! bang sibling. execute/3 is barely more than a delegation:

def execute(query, params \\ [], opts \\ []) do
  assert_enabled!()
  merged_opts = merge_opts(opts)

  Xandra.Cluster.execute(@cluster, query, params, merged_opts)
end

execute/3 vs execute!/3 is the usual Elixir split, and I pick per call site by who's asking. EntryReads.record/5 uses the tuple form so a Cassandra hiccup logs a warning and returns {:error, reason} instead of taking down the request that triggered it — read history is best-effort telemetry, not something worth 500ing a page over. The health check, by contrast, is fine to let errors bubble as tuples too. I reserve the bang variants for setup and iex, where a raise is exactly the feedback I want.

Prepared statements and caching

prepare/2 is worth a note because it's easy to over-engineer. I did not build a prepared-statement cache in this module — Xandra already keeps one, keyed per connection, and it re-prepares automatically when a connection is recycled. That last part matters: it means callers can hold onto a Xandra.Prepared.t() across reconnections without babysitting it. So the "cache" here is Xandra's internal :persistent_term-backed store, not something I reinvented on top. The right amount of code for a solved problem is no code.

The switch: DisabledError and defaulting to off

Here's the part I'm most pleased with. Cassandra is disabled by default. In config/test.exs it's a flat enabled: false, and in runtime.exs it defaults off in production too, because there is currently no write path with real callers — starting a Xandra pool that nothing writes to is pure liability.

Every public function starts with assert_enabled!():

defp assert_enabled! do
  unless enabled?() do
    raise DisabledError
  end
end

So in disabled mode the module raises BoringRSS.Cassandra.DisabledError at the call site rather than crashing the whole application at startup because a container isn't running. The test suite never starts a cluster; local dev turns it on with an env var once the container is up. This is the pattern that makes a third datastore tolerable: it's inert until you deliberately wake it.

Proving it's alive: cassandra_stats/0

The admin stats page needs a health readout, and BoringRSS.AnalyticsStats.cassandra_stats/0 provides it. It first gates on BoringRSS.Cassandra.enabled?() and short-circuits to {:error, :disabled} if off, then queries the system.* tables every node exposes without auth. system.local gives version, cluster name, DC, host ID; system.peers gives peer count; and it reads the schema straight from system_schema.keyspaces:

stmt = """
SELECT keyspace_name, durable_writes, replication
FROM system_schema.keyspaces
"""

The whole function is written to degrade rather than raise — each sub-query is wrapped so a failure returns an empty list or an error tuple, and the stats page renders "unavailable" instead of exploding. That's the same philosophy as the rest of the integration: the third database is allowed to be absent, and nothing else is allowed to care.

Would I do it again?

For this one data shape — high write volume, single-partition reads, self-expiring — yes. The honest caveat is that the write path is still gated off in production, so today Cassandra earns its keep mostly as designed-and-ready infrastructure rather than load-bearing. The discipline that makes that acceptable is the default-off switch: I got to build and wire the whole thing without paying the operational tax until there's traffic to justify it. If you're tempted by a third store, steal that part first.

All posts RSS Feed