Skip to content
Yareta Docs

Operations Runbook

Procedures for common operational conditions. Each entry: symptom → likely cause(s) → what to do. For active incidents needing severity triage and communication, see the Incident Response Guide; this document is the “what do I actually do” reference.

1. A Submission’s AI Processing Appears Stuck

Check, in order:

  1. /admin/jobs — is the job queued, running, or failed?
  2. The organization’s credit balance (/admin/organizations/[id] or getUserCreditStatusQuery) — jobs are skipped, not retried, when credits are insufficient (QueueBillingService.checkCreditsAndPlan). This is the single most common cause of a submission that never progresses. Top up or adjust credits, then manually re-trigger the relevant mutation (e.g. bulkTriggerFounderDna) if the job was skipped rather than queued.
  3. Cloudflare dashboard → Workers Logs for the queue worker — look for runWithD1Retry exhaustion (D1 overload) or an LLM-provider error that exhausted retryWithBackoffWhile.
  4. Confirm the relevant llm_services row (Admin → LLM Services) points at a currently-healthy provider — a misconfigured or rate-limited provider will fail every job routed to it.

2. Embeddings / RAG Search Results Look Stale or Missing

The submission-embedding cron sync is known to be paused in production (see the Integration Documentation) due to an unresolved auth gap between the cron worker and the app’s embed-all route. If a newly created or updated submission isn’t showing up in RAG/chat search:

  1. Confirm whether the cron is actually enabled in the current queue worker config (src/queue/wrangler-configs/production/wrangler.toml, [triggers]).
  2. If it’s still paused, trigger embedding manually via POST /api/admin/submissions/:id/embed-all (requires an authenticated admin session) rather than waiting for the cron.
  3. If re-enabling the cron permanently, first resolve the auth gap (add a service-token header to the cron’s outbound call) — otherwise every cron-triggered run will 401 silently.

3. A Third-Party Integration Sync Isn’t Working (Airtable / HubSpot / Attio / Typeform)

  1. Check whether the integration’s OAuth token has expired or been revoked on the third-party side — integrationAccounts stores the token but there’s no automated expiry-alerting found in the codebase.
  2. For Airtable specifically: webhook subscriptions expire and must be refreshed — check whether run-airtable-webhook-refresh-cron.ts (a daily cron) is running; a missed refresh is a plausible cause of silently stopped inbound sync.
  3. For webhook-driven integrations (Airtable, HubSpot, Typeform, Fireflies, Recall.ai, DocuSign), confirm the external platform is actually delivering webhooks (check the integration’s own dashboard/delivery logs on their side) before assuming the fault is on Yareta’s side.
  4. Check the relevant queue (processAirtableWebhookQueue, processHubspotWebhookQueue, processTypeformWebhookQueue) for a backlog or repeated failures.

4. Reports (MDD / Team Summary) Contain Malformed Text (Stray Markdown, Wrong Names, Truncated Content)

This class of issue traces back to the prompt, not a code bug, in most observed cases:

  1. Check whether the live system_prompts DB row for the affected prompt type matches the current code-side prompt (src/prompts/*.ts) — see Admin Guide §2 for why these can drift. A stale DB prompt missing a formatting or name-resolution instruction present in code is a known root cause of past incidents.
  2. If the DB prompt is current and the issue persists, check whether the affected content type has a defensive post-processing sanitizer (e.g. sanitizeProse() for team summaries) that might need extending for a new failure pattern — LLM instruction-following on formatting constraints has proven unreliable enough that some sections rely on code-level scrubbing rather than the prompt alone.
  3. For truncated JSON output specifically, check MODEL_MAX_OUTPUT_TOKENS/resolveCompletionTokenBudget for the model in use — this has previously caused a smaller model to silently truncate a large structured response (see ADR-004 and the CRITICAL_SERVICE_DEFAULTS override it prompted).

5. Credit Balance Looks Wrong (Under- or Over-Charged)

  1. Query llm_usage_log for the submission/feature in question, filtering on runId — this dedupes retries of the same logical request; multiple rows with the same runId and billedAt set on only one indicate the dedup worked correctly.
  2. Check creditUsageHistory for the idempotencyKey associated with the charge — a duplicate charge should never occur if the idempotency key matches an existing row.
  3. If a job appears to have consumed a full charge but produced no output, check for a FLAT_PLACEHOLDER_MARKER charge that was never superseded by a real usage-based charge (see the Technical Design doc) — this indicates the job started billing but never completed the LLM call.
  4. If a balance was drained to zero unexpectedly, check for a drainBalanceToZero event around the same timestamp — this is intentional behavior when a guarded deduction is refused due to a concurrent job, not a bug, but a cluster of these events close together may indicate two jobs racing on the same org’s credits and warrants investigation.

6. Deploy Succeeded but Behavior Didn’t Change

  1. Confirm which unit actually needed to change — a frontend/API change requires the Pages app deploy; a background-job change requires the queue worker deploy (scripts/queue-deploy.sh) — these are separate steps in ci-cd.yml and one can succeed while the other fails silently (no deploy-failure notification exists — see the Observability doc). Check both steps’ logs in the GitHub Actions run.
  2. If the change was to a prompt template in code, remember it also requires a DB sync (§4 above / Admin Guide §2) — a code deploy alone does not change live LLM behavior for prompt-template changes.
  3. Check whether the change was to an .env.example-listed variable that needs to be set as a dashboard Secret/Variable on the relevant Cloudflare project — wrangler.toml/CI only injects a specific known set of vars (NEXT_PUBLIC_APP_URL, GRAPHQL_ENDPOINT, GRAPHQL_TOKEN, QUEUE_API); most other env vars must be set directly on the Cloudflare dashboard for both the Pages project and, separately, the queue Worker (keep_vars = true means Worker secrets persist across deploys but must be set at least once).

7. D1 “Overloaded” or Transient Errors Under Load

This is a documented, expected D1 behavior under concurrent write pressure, not necessarily a bug. runWithD1Retry already retries these automatically (3 attempts, linear backoff). If retries are exhausting:

  1. Check whether a recent change increased max_concurrency on a D1-heavy queue beyond what production is tuned for (compare against src/queue/wrangler-configs/production/wrangler.toml’s deliberately conservative settings, e.g. dnaSignalsQueue at max_concurrency = 1).
  2. Check for an unusually large batch of jobs enqueued at once (e.g. a bulk bulkTriggerFounderDna call) that’s exceeding the per-queue tuning.

Related documents: Incident Response Guide · Troubleshooting Guide · Admin Guide · Observability & Monitoring