Maintenance

The jobs

Method	What it does	Auto-triggered?
runLibrarian(entityId, { promptOverride? })	Turns recent events into facts and tasks	With runPendingMaintenance(), or every autoLibrarianThreshold events (default 20) in 'auto' mode
runHeal(entityId, { promptOverride? })	Resolves contradictions, downgrades stale inferences, removes obsolete facts	With runPendingMaintenance(), or every autoHealThreshold events (default 100) in 'auto' mode
runReembed(entityId?, { skipExisting?, force? })	Computes missing or outdated embeddings	No
runPrune(entityId, opts)	Hard-deletes aged soft-deleted rows and old events	No
runOntologyBackfill(entityId)	Types untyped facts	No

Jobs for the same entity are mutually exclusive. A conflicting job throws WikiBusyError, which is safe to catch and retry later.

Librarian strategies

config: { librarian: { strategy: 'legacy' } } // default: 'ops'

	'legacy'	'ops' (default)
Input	The newest events, plus existing facts in the prompt	Events since the last pass only
Dedup	Done by the model	An embedding gate, with no LLM call: identical facts are a no-op, novel facts are added, and only similar-but-different candidates go to the model
Changes	Rewrites facts	ADD / UPDATE / SUPERSEDE / NOOP, in one batched call. Supersede keeps history.
Prompt overrides	prompts.librarianSystemPrompt	Ignored (ops uses its own schema)

Gate thresholds (gate: { k: 5, dupThreshold: 0.89, novelThreshold: 0.30 }) depend on your embedding model. The defaults were calibrated on BGE-small and are provisional for other models. Without embed, the gate uses keyword search, and every match goes to the model.

Known issue: ops with maintenance: 'auto' (fixed in 8.0.5)

From 7.8.0 through 8.0.4, strategy: 'ops' with maintenance: 'auto' skipped the first batch of events the first automatic pass should have read. With the count trigger that is autoLibrarianThreshold events (20 by default); with autoLibrarianTokenThreshold, it is the events that crossed the token threshold. The batch is counted from where the legacy librarian had stopped, if it had run before. It happened both on new entities and on entities upgraded from the legacy strategy. The pass made no model call for them, so they never became facts. The default maintenance: 'deferred' was not affected.

The skipped events are still in the event log unless they were pruned. After you upgrade, call write() again for the affected events to bring them into memory. Those are the first batch that crossed the threshold (autoLibrarianThreshold events, or the events that crossed autoLibrarianTokenThreshold): on a new entity from its first event, on an upgraded entity from the point where the legacy librarian stopped (some may predate the switch to ops). Each call adds a new event; the original rows stay in the log. Re-ingesting documents does not restore them, because ingest() does not write to the event log.

If an entity pruned events under the legacy strategy with maintenance: 'auto' and then switched to ops, the first ops pass also skipped as many of the post-switch events as were pruned. The fix also stops prune from causing this; for entities already affected, re-write those events in the same way.

Merge imports and the librarian position

A merge import re-dates events that would sort behind the destination’s librarian position, so the next pass reads them. Every import, merge: false included, re-stamps events whose date is unusable. Bundles from exportDump() and getMemoryBundle() carry the source’s librarian_watermark; events at or before it were consolidated at the source and keep their dates. If you split a bundle into chunks, copy librarian_watermark onto a chunk only if that chunk also carries the source’s facts; otherwise omit it. A chunk that keeps the watermark but drops the facts makes the importer treat its older events as already read, so they are never extracted and a later prune deletes them. A source running the legacy librarian (librarian.strategy: 'legacy') advances its librarian_watermark past events it never read, because a legacy pass reads only the newest 50 events. The importer trusts the watermark, so remove it from a bundle unless you know the source consolidated everything up to it.

Heal

Heal looks at candidates such as orphaned facts (no source for orphanAfterDays, default 30) and inferred facts older than staleInferredAfterDays (default 60). It asks the model to downgrade, delete, or replace them. immutable_document facts are never rewritten. Set either threshold to null to turn that check off.

Reembed

const r = await wiki.runReembed();          // all entities
await wiki.runReembed('user-123', { skipExisting: true }); // only facts with no vector
// → { embedded, skipped, failed, deferred, permanentlyFailed }

• Run it after you add embed() or change embedding models.
• deferred rows are inside an exponential backoff (starting at 60 s, doubling, capped at 24 h). That’s not an error.
• permanentlyFailed covers a float32_overflow failure or 5 failed attempts. Retry them with { force: true }, which also skips the backoff window for deferred rows.
• A convergence loop should test failed, not deferred.
• A storage_error counts in failed but never enters backoff, because storage is treated as a separate failure domain from the embedding provider. While writes keep failing, every sweep retries every row, so a loop on failed can spin. Treat repeated storage_error failures as an infrastructure problem to fix, not something to wait out.

Prune

await wiki.runPrune('user-123', {
  retainSoftDeletedFor: 7, // days; null to skip
  retainEventsFor: 30,     // days; null to skip
  vacuum: false,           // true rewrites the whole DB file (slow on mobile)
});
// → { entries, tasks, events }

Under 'deferred' maintenance or the 'ops' strategy, prune never deletes events the librarian hasn’t processed yet. It emits event_retention_held instead. To delete an entity’s events on request, read or not, use forget(entityId, { clearAll: true, includeEvents: true }).

Deferred maintenance

Since 8.0.0, write() only records the event by default. Nothing runs in the background and no model is called until you run maintenance, so no facts appear from events until you do. To have write() start the librarian and heal as background promises, set maintenance: 'auto', but not on serverless or request-scoped hosts, where those promises can be frozen or killed partway through.

const wiki = new WikiMemory(db, {
  llmProvider,
  config: { maintenance: 'deferred', librarian: { strategy: 'ops' } },
});

await wiki.write(entityId, { event_type: 'observation', summary, occurred_at });

const report = await wiki.runPendingMaintenance({ tokenBudget: 20_000, deadlineMs: 8_000 });
report.stoppedReason.reason; // 'complete' | 'deadline' | 'budget_exhausted' | 'budget_too_small'
// Nothing is still running at this point, so it's safe to persist the DB file
// (unless you set drainExternalDeletionsOnSetup and deadlineMs cut short the wait for it:
// then await wiki.drain()).

• getPendingMaintenance() shows the backlog per entity without calling a model. It also reports pendingExternalDeletions when your vector index still owes removals and this process can act on them (a hook is configured, or forceDeleteIgnoreRankerHook is on). An entity with only owed removals is listed only while at least one of its rows is outside backoff (never attempted, or last attempted longer ago than min(2^attempts, 3600) seconds). That doesn’t make a loop over getPendingMaintenance() end on its own: each pass stops after 5 rejections in a row or one timeout, so the rows it didn’t try keep the entity listed.
• runPendingMaintenance() also delivers those owed removals first (the 'externalDeletions' job). That work makes no model call and doesn’t use the token budget. If you loop until nothing is pending, bound the iterations: a hook that keeps failing keeps removals pending, possibly for many passes.
• The budget counts text-generation tokens only, not embedding calls. The counts come from the provider when you implement generateTextWithUsage, and are estimated otherwise.
• Entities share the budget fairly: each round runs one batch per entity.
• A crash mid-batch loses nothing, because the librarian’s watermark only advances after a commit.
• Use 'deferred' together with 'ops'. The legacy strategy only reads the newest 50 events per pass, so a larger backlog would be skipped.

Shutting down in auto mode?

await wiki.drain() before your app is suspended or the process exits, so in-flight background jobs can finish.

On mobile: the Night Shift pattern

Phones are hard on background work: iOS can kill memory-hungry processes, and long LLM passes drain the battery. A pattern that works well is to turn off auto maintenance and run it in the foreground while the device is charging:

import * as Battery from 'expo-battery';
import { activateKeepAwakeAsync, deactivateKeepAwake } from 'expo-keep-awake';

const wiki = createWiki(db, {
  llmProvider,
  config: { maintenance: 'deferred', librarian: { strategy: 'ops' } },
});

export async function runNightShift() {
  const state = await Battery.getBatteryStateAsync();
  if (state !== Battery.BatteryState.CHARGING && state !== Battery.BatteryState.FULL) return;

  await activateKeepAwakeAsync();
  try {
    await wiki.runPendingMaintenance({ deadlineMs: 15 * 60_000 });
  } finally {
    deactivateKeepAwake();
  }
}

Curated Journal ships the same idea as its Night Shift mode: a full-screen maintenance session the user starts when the device is plugged in.

Live job status

wiki.getEntityStatus('user-42'); // { ingesting, librarian, heal }

const unsubscribe = wiki.subscribeEntityStatus('user-42', (s) => render(s));
unsubscribe();

The callback fires once, synchronously, before subscribeEntityStatus returns (treat it as the initial value), then again whenever any of the three flags flips.

• Each call may get a fresh object. Compare the three booleans, not object identity.
• A callback that throws is caught and logged. It doesn’t affect other subscribers or the job.
• A subscription covers one entity. There’s no wildcard form.
• unsubscribe() is safe to call more than once.

In React, use useEntityStatus.