Cluster Cache: Recovery and Reliability

Cluster Cache: Recovery and Reliability#

This page covers retention gaps, recovery, transaction ordering, deployment checklists, and production troubleshooting.

Retention, recovery, and offline nodes#

Each node stores (cluster, nodeId, lastEventId) in its local Node Cache SQLite database. The transport retains only a finite event history. If a node’s cursor predates the oldest available event, it cannot know every invalidation it missed. Before normal consumption, Cluster Cache therefore:

  1. clears that node’s entire local namespace;

  2. resets its cursor to the oldest retained event ID;

  3. resumes consumption on subsequent reads of the event stream.

recoverIfRequired() exposes this check directly and returns true when a clear/recovery occurred. Calling consume() already calls it, so direct calls are mainly useful for health checks or explicit startup handling.

This safe reset is why event retention is an availability and cache-warmth decision rather than a correctness shortcut. Short retention causes more full local clears after outages; long retention increases transport storage and catch-up work.

For PdoInvalidationTransport, prune events in small batches after a chosen retention period:

// Keep events for seven days. Run this repeatedly from maintenance.
$deleted = $transport->pruneBefore(time() - 7 * 24 * 60 * 60, 5_000);

The PDO transport’s pruneBefore() is intentionally not part of the generic transport interface. Other transport implementations should use their own retention mechanism.

Retention-pruning schedule#

Event pruning is separate from consumption. For the PDO transport, schedule a maintenance command against the shared database once; do not run competing pruners on every web node unless your job platform coordinates them.

// bin/cache-cluster-prune.php
$transport->pruneBefore(time() - 7 * 24 * 60 * 60, 5_000);

# One elected scheduler/worker, hourly; run repeatedly if the backlog is large.
12 * * * * www-data /usr/bin/php /srv/application/bin/cache-cluster-prune.php

Keep the retention window longer than the largest supported outage. Pruning more often does not permit a shorter retention period safely.

Failure ordering and transactional writes#

An invalidation involves two independent actions: local invalidation and event publication. invalidateLocallyFirst chooses which happens first.

With the default true:

  1. The initiating node becomes fresh immediately.

  2. If publication fails, remote nodes are not notified and may be stale until TTL or a later successful invalidation.

With false:

  1. Remote propagation is recorded before the initiating local clear.

  2. If local invalidation fails, the originating node may remain stale while remote nodes later become fresh.

Neither sequence is atomic across the source database and the event transport. For important database writes, use the PDO transactional-outbox helper with the same database connection:

use Infocyph\CacheLayer\Cluster\Event\InvalidationEvent;

$pdo->beginTransaction();
try {
    updateProduct($pdo, 42, $input);
    $transport->publishWithinTransaction(
        $pdo,
        InvalidationEvent::key(
            'production',
            'catalog',
            'product.42',
            gethostname() ?: 'catalog-unknown-node',
        ),
    );
    $pdo->commit();
} catch (Throwable $exception) {
    if ($pdo->inTransaction()) {
        $pdo->rollBack();
    }

    throw $exception;
}

// The event is now durable. Clear this origin node explicitly after commit.
$runtime->cache()->delete('product.42');

publishWithinTransaction() requires the exact PDO connection held by the transport and an active transaction. It makes the source update and event insertion atomic. It does not itself invalidate the origin node: origin events are skipped by that node’s consumer, so the explicit local delete after a successful commit is required. Treat a failure of that delete as operationally important and retry/alert; a bounded TTL still limits the stale window.

PdoInvalidationTransport implements TransactionalInvalidationTransportInterface. Framework/database integrations should depend on that interface rather than the concrete PDO class when they need to attach an invalidation event to a database transaction.

CacheLayer also provides a framework-neutral ClusterOutbox helper through $runtime->outbox($pdo). Create it within the source transaction, publish its invalidations alongside the source write, commit, then apply its local invalidation after the commit succeeds:

$pdo->beginTransaction();
try {
    updateProduct($pdo, 42, $input);
    $outbox = $runtime->outbox($pdo);
    $outbox->invalidateKey('product.42');
    $pdo->commit();
    $outbox->applyLocally();
} catch (Throwable $exception) {
    if ($pdo->inTransaction()) {
        $pdo->rollBack();
    }

    throw $exception;
}

The outbox event is deliberately replayed by its origin node as well. Therefore if the process fails after commit but before applyLocally(), the next local consumer run still invalidates the origin cache safely. A framework can wrap the final local application in its after-commit callback; transaction ownership and callback registration remain application concerns.

Operations checklist#

At deployment:

  • create a local, private cache directory on every node;

  • configure one shared durable transport per logical cluster;

  • give every node a unique ID and use the same cluster name;

  • start a consumer on every node before or alongside application traffic;

  • set event retention longer than the expected maximum outage;

  • choose bounded cache TTLs even with fast consumers.

During normal operation:

  • use $runtime->cache()->remember() for local reads;

  • use $runtime->cache()->set() only for local fills;

  • call runtime invalidateKey(), invalidateTag(), or clearNamespace() after committed source-data changes;

  • run a consumer on every node and node SQLite pruning/checkpoint maintenance separately;

  • run shared-transport retention pruning from one coordinated scheduler;

  • monitor consumer lag, transport errors, cursor age, cache hit rate, and retention-prune activity.

During an incident:

  • if a node is suspect, clearNamespace() gives a broad distributed reset;

  • if a node was offline longer than retention, allow its consumer recovery to clear its local namespace before it resumes;

  • if the transport is unavailable, ordinary local reads continue, but remote invalidations stop propagating—restore transport and process the backlog;

  • do not rely on Cluster Cache alone for an immediate security revoke; enforce that decision at the authoritative service or data store.

Troubleshooting#

A remote node still has an old value

Confirm the source change called a runtime invalidation method, not $runtime->cache()->delete() or invalidateTag(). Then verify the remote node’s consumer is running, connected to the same cluster and shared transport, and using the expected namespace.

A node repeatedly clears its cache on startup

Its cursor is likely behind event retention, its local SQLite file is being discarded, or its node ID changes every deployment. Inspect retention, volume persistence, and identity configuration.

Events are growing without bound

Configure transport retention. With the PDO transport, schedule bounded pruneBefore() calls. Do not prune more aggressively than your longest supported outage unless frequent full local-cache recovery is acceptable.

The origin node appears stale after a transactional outbox write

The consumer intentionally skips its own event. Ensure the application explicitly deletes or invalidates the origin node after the transaction commits, as shown above.

Can multiple clusters share one transport?

Yes. Use distinct cluster names. Events and cursors are partitioned by cluster name; capacity, retention, and operational monitoring remain shared concerns of the underlying transport.